Nov 01 22:20:32.570435 ip-10-0-142-238 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 01 22:20:32.570447 ip-10-0-142-238 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 01 22:20:32.570454 ip-10-0-142-238 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 01 22:20:32.570714 ip-10-0-142-238 systemd[1]: Failed to start Kubernetes Kubelet. Nov 01 22:20:43.852887 ip-10-0-142-238 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 01 22:20:43.852901 ip-10-0-142-238 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 0b5724562c6f4dacb38551e3e47396b1 -- Nov 01 22:22:39.693212 ip-10-0-142-238 systemd[1]: Starting Kubernetes Kubelet... Nov 01 22:22:40.157168 ip-10-0-142-238 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 22:22:40.157168 ip-10-0-142-238 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 22:22:40.157168 ip-10-0-142-238 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 22:22:40.157168 ip-10-0-142-238 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 22:22:40.163247 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163141 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:22:40.163247 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163242 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:22:40.163247 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163247 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:22:40.163247 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163252 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:22:40.163247 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163257 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:22:40.163561 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163267 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:22:40.163561 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163271 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:22:40.163561 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163275 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:22:40.163561 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163279 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:22:40.163561 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163283 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:22:40.163561 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163286 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:22:40.163561 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163290 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:22:40.163561 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163294 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:22:40.163561 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163298 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:22:40.163561 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163303 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:22:40.163561 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163306 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:22:40.163561 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163310 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:22:40.163561 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163314 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:22:40.163561 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163321 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:22:40.163561 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163325 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:22:40.163561 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163329 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:22:40.163561 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163334 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:22:40.163561 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163339 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:22:40.163561 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163343 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:22:40.163561 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163348 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:22:40.164388 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163353 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:22:40.164388 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163357 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:22:40.164388 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163364 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:22:40.164388 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163372 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:22:40.164388 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163378 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:22:40.164388 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163387 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:22:40.164388 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163392 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:22:40.164388 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163397 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:22:40.164388 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163401 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:22:40.164388 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163407 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:22:40.164388 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163412 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:22:40.164388 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163416 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:22:40.164388 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163421 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:22:40.164388 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163425 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:22:40.164388 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163430 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:22:40.164388 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163434 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:22:40.164388 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163439 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:22:40.164388 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163443 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:22:40.164388 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163453 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:22:40.165194 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163457 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:22:40.165194 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163462 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:22:40.165194 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163466 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:22:40.165194 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163471 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:22:40.165194 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163476 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:22:40.165194 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163480 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:22:40.165194 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163484 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:22:40.165194 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163489 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:22:40.165194 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163494 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:22:40.165194 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163498 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:22:40.165194 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163503 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:22:40.165194 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163507 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:22:40.165194 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163517 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:22:40.165194 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163521 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:22:40.165194 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163526 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:22:40.165194 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163530 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:22:40.165194 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163535 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:22:40.165194 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163540 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:22:40.165194 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163546 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:22:40.165194 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163550 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:22:40.166048 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163555 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:22:40.166048 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163559 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:22:40.166048 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163563 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:22:40.166048 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163567 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:22:40.166048 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163571 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:22:40.166048 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163579 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:22:40.166048 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163584 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:22:40.166048 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163588 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:22:40.166048 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163593 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:22:40.166048 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163597 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:22:40.166048 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163602 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:22:40.166048 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163606 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:22:40.166048 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163610 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:22:40.166048 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163615 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:22:40.166048 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163620 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:22:40.166048 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163624 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:22:40.166048 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163629 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:22:40.166048 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163633 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:22:40.166048 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163643 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:22:40.166888 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163647 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:22:40.166888 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163652 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:22:40.166888 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163656 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:22:40.166888 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163660 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:22:40.166888 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163664 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:22:40.166888 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163669 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:22:40.166888 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163674 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:22:40.166888 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163681 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:22:40.166888 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163686 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:22:40.166888 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163691 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:22:40.166888 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163700 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:22:40.166888 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163706 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:22:40.166888 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163711 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:22:40.166888 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163716 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:22:40.166888 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163721 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:22:40.166888 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163727 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:22:40.166888 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163733 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:22:40.166888 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163740 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:22:40.166888 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163745 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:22:40.166888 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163750 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:22:40.167773 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163755 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:22:40.167773 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163760 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:22:40.167773 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.163769 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:22:40.167773 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166170 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:22:40.167773 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166189 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:22:40.167773 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166194 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:22:40.167773 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166199 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:22:40.167773 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166204 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:22:40.167773 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166208 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:22:40.167773 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166213 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:22:40.167773 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166217 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:22:40.167773 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166222 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:22:40.167773 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166226 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:22:40.167773 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166230 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:22:40.167773 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166235 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:22:40.167773 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166239 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:22:40.167773 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166243 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:22:40.167773 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166252 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:22:40.167773 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166257 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:22:40.168610 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166262 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:22:40.168610 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166270 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:22:40.168610 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166274 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:22:40.168610 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166278 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:22:40.168610 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166285 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:22:40.168610 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166297 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:22:40.168610 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166302 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:22:40.168610 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166306 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:22:40.168610 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166311 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:22:40.168610 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166315 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:22:40.168610 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166324 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:22:40.168610 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166329 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:22:40.168610 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166333 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:22:40.168610 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166338 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:22:40.168610 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166342 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:22:40.168610 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166346 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:22:40.168610 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166351 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:22:40.168610 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166355 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:22:40.168610 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166360 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:22:40.169398 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166364 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:22:40.169398 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166369 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:22:40.169398 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166373 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:22:40.169398 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166378 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:22:40.169398 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166386 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:22:40.169398 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166391 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:22:40.169398 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166396 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:22:40.169398 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166400 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:22:40.169398 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166404 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:22:40.169398 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166409 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:22:40.169398 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166414 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:22:40.169398 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166418 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:22:40.169398 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166422 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:22:40.169398 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166427 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:22:40.169398 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166431 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:22:40.169398 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166435 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:22:40.169398 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166439 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:22:40.169398 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166449 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:22:40.169398 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166453 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:22:40.169876 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166465 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:22:40.169876 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166470 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:22:40.169876 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166475 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:22:40.169876 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166479 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:22:40.169876 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166483 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:22:40.169876 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166487 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:22:40.169876 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166492 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:22:40.169876 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166497 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:22:40.169876 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166508 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:22:40.169876 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166521 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:22:40.169876 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166525 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:22:40.169876 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166529 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:22:40.169876 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166534 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:22:40.169876 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166539 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:22:40.169876 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166543 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:22:40.169876 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166547 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:22:40.169876 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166552 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:22:40.169876 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166556 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:22:40.169876 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166560 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:22:40.169876 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166565 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:22:40.170356 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166569 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:22:40.170356 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166573 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:22:40.170356 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166582 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:22:40.170356 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166586 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:22:40.170356 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166591 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:22:40.170356 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166595 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:22:40.170356 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166599 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:22:40.170356 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166603 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:22:40.170356 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166608 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:22:40.170356 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166612 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:22:40.170356 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166616 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:22:40.170356 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166620 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:22:40.170356 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166625 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:22:40.170356 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166637 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:22:40.170356 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166642 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:22:40.170356 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166650 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:22:40.170356 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166655 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:22:40.170356 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166659 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:22:40.170356 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166663 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:22:40.170356 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166667 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:22:40.170842 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166672 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:22:40.170842 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166676 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:22:40.170842 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166680 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:22:40.170842 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166711 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:22:40.170842 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166719 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:22:40.170842 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166725 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:22:40.170842 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166730 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:22:40.170842 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166740 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:22:40.170842 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166744 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:22:40.170842 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166749 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:22:40.170842 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166754 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:22:40.170842 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.166761 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:22:40.170842 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168329 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:22:40.170842 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168340 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:22:40.170842 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168350 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:22:40.170842 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168356 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:22:40.170842 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168361 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:22:40.170842 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168365 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:22:40.170842 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168370 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:22:40.170842 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168374 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:22:40.171333 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168378 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:22:40.171333 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168383 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:22:40.171333 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168387 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:22:40.171333 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168391 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:22:40.171333 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168396 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:22:40.171333 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168408 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:22:40.171333 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168418 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:22:40.171333 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168422 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:22:40.171333 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168427 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:22:40.171333 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168431 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:22:40.171333 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168436 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:22:40.171333 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168440 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:22:40.171333 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168444 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:22:40.171333 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168449 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:22:40.171333 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168453 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:22:40.171333 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168458 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:22:40.171333 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168463 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:22:40.171333 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168467 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:22:40.171333 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168472 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:22:40.171333 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168481 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:22:40.171801 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168485 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:22:40.171801 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168489 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:22:40.171801 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168494 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:22:40.171801 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168498 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:22:40.171801 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168502 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:22:40.171801 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168507 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:22:40.171801 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168511 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:22:40.171801 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168515 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:22:40.171801 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168519 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:22:40.171801 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168524 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:22:40.171801 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168528 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:22:40.171801 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168532 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:22:40.171801 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168541 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:22:40.171801 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168546 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:22:40.171801 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168551 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:22:40.171801 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168558 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:22:40.171801 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168564 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:22:40.171801 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168569 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:22:40.171801 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168583 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:22:40.172262 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168588 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:22:40.172262 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168593 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:22:40.172262 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168597 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:22:40.172262 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168601 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:22:40.172262 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168605 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:22:40.172262 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168614 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:22:40.172262 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168618 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:22:40.172262 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168622 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:22:40.172262 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168626 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:22:40.172262 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168631 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:22:40.172262 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168635 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:22:40.172262 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168639 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:22:40.172262 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168644 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:22:40.172262 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168648 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:22:40.172262 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168653 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:22:40.172262 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168657 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:22:40.172262 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168661 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:22:40.172262 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168666 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:22:40.172262 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168674 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:22:40.172752 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168678 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:22:40.172752 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168682 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:22:40.172752 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168687 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:22:40.172752 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168693 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:22:40.172752 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168697 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:22:40.172752 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168702 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:22:40.172752 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168706 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:22:40.172752 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168730 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:22:40.172752 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.168854 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:22:40.172752 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.169034 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:22:40.172752 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.169040 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:22:40.172752 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.169043 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:22:40.172752 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.169046 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:22:40.172752 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.169049 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:22:40.172752 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.169057 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:22:40.172752 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.169063 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:22:40.172752 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.169066 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:22:40.172752 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.169070 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:22:40.172752 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.169073 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:22:40.173228 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.169076 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:22:40.173228 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.169079 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:22:40.173228 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.169082 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:22:40.173228 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.169085 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:22:40.173228 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.169088 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:22:40.173228 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.169090 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:22:40.173228 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.169093 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:22:40.173228 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.169096 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:22:40.173228 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.169102 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:22:40.173228 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.169104 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:22:40.173228 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.169107 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:22:40.173228 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.169110 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:22:40.173228 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.169112 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:22:40.173228 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.169115 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:22:40.173228 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.169118 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:22:40.173228 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.169120 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:22:40.173228 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.169123 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:22:40.173228 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.169125 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:22:40.173228 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.169128 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:22:40.173228 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.169130 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:22:40.174615 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.169133 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:22:40.174615 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171115 2576 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 22:22:40.174615 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171129 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 22:22:40.174615 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171141 2576 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 22:22:40.174615 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171146 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 22:22:40.174615 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171151 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 22:22:40.174615 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171154 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 22:22:40.174615 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171160 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 22:22:40.174615 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171164 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 22:22:40.174615 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171167 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 22:22:40.174615 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171170 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 22:22:40.174615 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171174 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 22:22:40.174615 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171177 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 22:22:40.174615 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171180 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 22:22:40.174615 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171183 2576 flags.go:64] FLAG: --cgroup-root="" Nov 01 22:22:40.174615 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171186 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 22:22:40.174615 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171189 2576 flags.go:64] FLAG: --client-ca-file="" Nov 01 22:22:40.174615 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171192 2576 flags.go:64] FLAG: --cloud-provider="external" Nov 01 22:22:40.174615 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171196 2576 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 22:22:40.174615 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171202 2576 flags.go:64] FLAG: --cluster-domain="" Nov 01 22:22:40.174615 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171205 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 22:22:40.174615 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171208 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 22:22:40.174615 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171212 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 22:22:40.174615 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171216 2576 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 22:22:40.175213 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171227 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 22:22:40.175213 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171230 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 22:22:40.175213 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171233 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 22:22:40.175213 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171237 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 22:22:40.175213 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171240 2576 flags.go:64] FLAG: --contention-profiling="false" Nov 01 22:22:40.175213 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171243 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 22:22:40.175213 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171246 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 22:22:40.175213 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171249 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 22:22:40.175213 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171252 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 22:22:40.175213 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171256 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 22:22:40.175213 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171260 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 22:22:40.175213 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171263 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 22:22:40.175213 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171266 2576 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 22:22:40.175213 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171269 2576 flags.go:64] FLAG: --enable-server="true" Nov 01 22:22:40.175213 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171272 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 22:22:40.175213 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171276 2576 flags.go:64] FLAG: --event-burst="100" Nov 01 22:22:40.175213 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171280 2576 flags.go:64] FLAG: --event-qps="50" Nov 01 22:22:40.175213 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171283 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 22:22:40.175213 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171286 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 22:22:40.175213 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171288 2576 flags.go:64] FLAG: --eviction-hard="" Nov 01 22:22:40.175213 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171293 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 22:22:40.175213 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171296 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 22:22:40.175213 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171299 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 22:22:40.175213 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171302 2576 flags.go:64] FLAG: --eviction-soft="" Nov 01 22:22:40.175213 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171304 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 22:22:40.175820 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171307 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 22:22:40.175820 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171310 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 22:22:40.175820 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171314 2576 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 22:22:40.175820 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171317 2576 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 22:22:40.175820 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171320 2576 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 22:22:40.175820 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171323 2576 flags.go:64] FLAG: --feature-gates="" Nov 01 22:22:40.175820 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171327 2576 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 22:22:40.175820 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171330 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 22:22:40.175820 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171333 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 22:22:40.175820 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171336 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 22:22:40.175820 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171340 2576 flags.go:64] FLAG: --healthz-port="10248" Nov 01 22:22:40.175820 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171343 2576 flags.go:64] FLAG: --help="false" Nov 01 22:22:40.175820 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171346 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:22:40.175820 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171355 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 22:22:40.175820 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171359 2576 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 22:22:40.175820 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171362 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 22:22:40.175820 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171368 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 22:22:40.175820 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171371 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 22:22:40.175820 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171374 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 22:22:40.175820 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171377 2576 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 22:22:40.175820 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171380 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 22:22:40.175820 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171383 2576 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 22:22:40.175820 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171386 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 22:22:40.176412 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171389 2576 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 22:22:40.176412 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171392 2576 flags.go:64] FLAG: --kube-reserved="" Nov 01 22:22:40.176412 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171395 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 22:22:40.176412 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171398 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 22:22:40.176412 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171401 2576 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 22:22:40.176412 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171404 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 22:22:40.176412 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171406 2576 flags.go:64] FLAG: --lock-file="" Nov 01 22:22:40.176412 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171409 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 22:22:40.176412 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171412 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 22:22:40.176412 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171415 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 22:22:40.176412 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171421 2576 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 22:22:40.176412 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171424 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 22:22:40.176412 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171427 2576 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 22:22:40.176412 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171429 2576 flags.go:64] FLAG: --logging-format="text" Nov 01 22:22:40.176412 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171432 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 22:22:40.176412 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171436 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 22:22:40.176412 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171438 2576 flags.go:64] FLAG: --manifest-url="" Nov 01 22:22:40.176412 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171441 2576 flags.go:64] FLAG: --manifest-url-header="" Nov 01 22:22:40.176412 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171446 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 22:22:40.176412 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171449 2576 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 22:22:40.176412 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171453 2576 flags.go:64] FLAG: --max-pods="110" Nov 01 22:22:40.176412 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171456 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 22:22:40.176412 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171459 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 22:22:40.176412 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171467 2576 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 22:22:40.176412 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171470 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 22:22:40.177041 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171475 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 22:22:40.177041 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171478 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 22:22:40.177041 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171481 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 22:22:40.177041 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171490 2576 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 22:22:40.177041 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171493 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 22:22:40.177041 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171496 2576 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 22:22:40.177041 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171499 2576 flags.go:64] FLAG: --pod-cidr="" Nov 01 22:22:40.177041 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171502 2576 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 22:22:40.177041 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171505 2576 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 22:22:40.177041 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171508 2576 flags.go:64] FLAG: --pods-per-core="0" Nov 01 22:22:40.177041 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171511 2576 flags.go:64] FLAG: --port="10250" Nov 01 22:22:40.177041 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171514 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 22:22:40.177041 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171517 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0a8a9e93cb5c0d220" Nov 01 22:22:40.177041 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171520 2576 flags.go:64] FLAG: --qos-reserved="" Nov 01 22:22:40.177041 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171522 2576 flags.go:64] FLAG: --read-only-port="10255" Nov 01 22:22:40.177041 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171526 2576 flags.go:64] FLAG: --register-node="true" Nov 01 22:22:40.177041 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171529 2576 flags.go:64] FLAG: --register-with-taints="" Nov 01 22:22:40.177041 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171532 2576 flags.go:64] FLAG: --registry-burst="10" Nov 01 22:22:40.177041 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171535 2576 flags.go:64] FLAG: --registry-qps="5" Nov 01 22:22:40.177041 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171538 2576 flags.go:64] FLAG: --reserved-cpus="" Nov 01 22:22:40.177041 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171541 2576 flags.go:64] FLAG: --reserved-memory="" Nov 01 22:22:40.177041 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171545 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 22:22:40.177041 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171548 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 22:22:40.177041 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171551 2576 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 22:22:40.177041 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171553 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 22:22:40.177041 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171556 2576 flags.go:64] FLAG: --runonce="false" Nov 01 22:22:40.177693 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171559 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 22:22:40.177693 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171562 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 22:22:40.177693 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171565 2576 flags.go:64] FLAG: --seccomp-default="false" Nov 01 22:22:40.177693 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171568 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 22:22:40.177693 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171571 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 22:22:40.177693 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171580 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 22:22:40.177693 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171585 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 22:22:40.177693 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171588 2576 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 22:22:40.177693 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171591 2576 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 22:22:40.177693 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171593 2576 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 22:22:40.177693 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171596 2576 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 22:22:40.177693 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171599 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 22:22:40.177693 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171602 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 22:22:40.177693 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171605 2576 flags.go:64] FLAG: --system-cgroups="" Nov 01 22:22:40.177693 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171607 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 22:22:40.177693 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171614 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 22:22:40.177693 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171618 2576 flags.go:64] FLAG: --tls-cert-file="" Nov 01 22:22:40.177693 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171620 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 22:22:40.177693 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171627 2576 flags.go:64] FLAG: --tls-min-version="" Nov 01 22:22:40.177693 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171630 2576 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 22:22:40.177693 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171633 2576 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 22:22:40.177693 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171635 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 22:22:40.177693 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171638 2576 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 22:22:40.177693 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171641 2576 flags.go:64] FLAG: --v="2" Nov 01 22:22:40.177693 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171646 2576 flags.go:64] FLAG: --version="false" Nov 01 22:22:40.178309 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171651 2576 flags.go:64] FLAG: --vmodule="" Nov 01 22:22:40.178309 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171655 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 22:22:40.178309 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.171658 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 22:22:40.178309 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171820 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:22:40.178309 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171824 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:22:40.178309 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171842 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:22:40.178309 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171846 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:22:40.178309 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171850 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:22:40.178309 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171854 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:22:40.178309 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171857 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:22:40.178309 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171860 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:22:40.178309 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171862 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:22:40.178309 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171865 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:22:40.178309 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171872 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:22:40.178309 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171875 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:22:40.178309 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171877 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:22:40.178309 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171880 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:22:40.178309 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171882 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:22:40.178309 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171885 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:22:40.178309 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171888 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:22:40.178814 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171890 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:22:40.178814 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171893 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:22:40.178814 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171895 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:22:40.178814 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171898 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:22:40.178814 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171900 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:22:40.178814 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171903 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:22:40.178814 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171905 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:22:40.178814 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171908 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:22:40.178814 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171911 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:22:40.178814 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171913 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:22:40.178814 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171916 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:22:40.178814 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171918 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:22:40.178814 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171921 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:22:40.178814 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171923 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:22:40.178814 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171926 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:22:40.178814 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171928 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:22:40.178814 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171931 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:22:40.178814 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171933 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:22:40.178814 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171936 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:22:40.178814 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171938 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:22:40.179315 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171941 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:22:40.179315 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171945 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:22:40.179315 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171948 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:22:40.179315 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171950 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:22:40.179315 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171953 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:22:40.179315 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171956 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:22:40.179315 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171959 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:22:40.179315 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171962 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:22:40.179315 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171965 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:22:40.179315 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171967 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:22:40.179315 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171970 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:22:40.179315 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171972 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:22:40.179315 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171975 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:22:40.179315 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171978 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:22:40.179315 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171980 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:22:40.179315 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171984 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:22:40.179315 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171987 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:22:40.179315 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171990 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:22:40.179315 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171993 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:22:40.179762 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171995 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:22:40.179762 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.171997 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:22:40.179762 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172000 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:22:40.179762 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172002 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:22:40.179762 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172005 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:22:40.179762 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172007 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:22:40.179762 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172010 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:22:40.179762 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172012 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:22:40.179762 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172014 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:22:40.179762 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172017 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:22:40.179762 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172020 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:22:40.179762 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172022 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:22:40.179762 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172025 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:22:40.179762 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172027 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:22:40.179762 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172031 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:22:40.179762 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172033 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:22:40.179762 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172036 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:22:40.179762 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172038 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:22:40.179762 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172042 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:22:40.179762 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172044 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:22:40.180255 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172049 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:22:40.180255 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172051 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:22:40.180255 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172054 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:22:40.180255 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172057 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:22:40.180255 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172059 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:22:40.180255 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172062 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:22:40.180255 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172064 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:22:40.180255 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172066 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:22:40.180255 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172069 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:22:40.180255 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172071 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:22:40.180255 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172074 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:22:40.180255 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172076 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:22:40.180255 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172079 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:22:40.180255 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172081 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:22:40.180255 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172084 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:22:40.180255 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172086 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:22:40.180255 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172089 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:22:40.180255 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172091 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:22:40.180255 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172094 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:22:40.180697 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172096 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:22:40.180697 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172099 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:22:40.180697 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172101 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:22:40.180697 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172103 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:22:40.180697 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172106 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:22:40.180697 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172108 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:22:40.180697 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172111 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:22:40.180697 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172114 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:22:40.180697 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172117 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:22:40.180697 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172119 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:22:40.180697 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172122 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:22:40.180697 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.172151 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 22:22:40.180697 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172193 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:22:40.180697 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172198 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:22:40.180697 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172202 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:22:40.180697 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172205 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:22:40.181101 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172207 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:22:40.181101 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172210 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:22:40.181101 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172213 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:22:40.181101 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172217 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:22:40.181101 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172221 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:22:40.181101 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172224 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:22:40.181101 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172226 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:22:40.181101 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172229 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:22:40.181101 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172231 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:22:40.181101 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172234 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:22:40.181101 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172237 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:22:40.181101 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172239 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:22:40.181101 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172242 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:22:40.181101 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172244 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:22:40.181101 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172247 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:22:40.181101 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172249 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:22:40.181101 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172252 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:22:40.181101 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172254 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:22:40.181101 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172257 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:22:40.181546 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172260 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:22:40.181546 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172263 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:22:40.181546 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172266 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:22:40.181546 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172268 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:22:40.181546 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172273 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:22:40.181546 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172276 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:22:40.181546 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172278 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:22:40.181546 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172281 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:22:40.181546 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172283 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:22:40.181546 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172287 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:22:40.181546 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172290 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:22:40.181546 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172293 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:22:40.181546 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172296 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:22:40.181546 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172299 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:22:40.181546 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172302 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:22:40.181546 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172304 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:22:40.181546 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172307 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:22:40.181546 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172310 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:22:40.181546 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172312 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:22:40.182015 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172315 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:22:40.182015 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172317 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:22:40.182015 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172320 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:22:40.182015 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172322 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:22:40.182015 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172325 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:22:40.182015 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172327 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:22:40.182015 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172329 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:22:40.182015 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172332 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:22:40.182015 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172334 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:22:40.182015 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172337 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:22:40.182015 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172339 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:22:40.182015 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172342 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:22:40.182015 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172344 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:22:40.182015 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172347 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:22:40.182015 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172349 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:22:40.182015 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172351 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:22:40.182015 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172354 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:22:40.182015 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172358 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:22:40.182015 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172361 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:22:40.182015 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172364 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:22:40.182509 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172366 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:22:40.182509 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172369 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:22:40.182509 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172374 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:22:40.182509 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172378 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:22:40.182509 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172381 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:22:40.182509 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172383 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:22:40.182509 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172386 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:22:40.182509 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172389 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:22:40.182509 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172391 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:22:40.182509 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172394 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:22:40.182509 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172396 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:22:40.182509 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172399 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:22:40.182509 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172401 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:22:40.182509 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172404 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:22:40.182509 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172407 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:22:40.182509 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172409 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:22:40.182509 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172411 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:22:40.182509 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172414 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:22:40.182509 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172416 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:22:40.182509 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172419 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:22:40.183077 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172422 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:22:40.183077 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172424 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:22:40.183077 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172427 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:22:40.183077 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172429 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:22:40.183077 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172432 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:22:40.183077 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172434 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:22:40.183077 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172437 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:22:40.183077 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172439 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:22:40.183077 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172442 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:22:40.183077 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172445 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:22:40.183077 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172448 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:22:40.183077 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172450 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:22:40.183077 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172453 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:22:40.183077 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172455 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:22:40.183077 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172459 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:22:40.183077 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172462 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:22:40.183077 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172466 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:22:40.183077 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172469 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:22:40.183077 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172471 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:22:40.183077 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172474 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:22:40.183579 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172477 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:22:40.183579 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172479 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:22:40.183579 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172481 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:22:40.183579 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.172484 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:22:40.183579 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.180389 2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 22:22:40.183579 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.180410 2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 22:22:40.183579 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.180427 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 22:22:40.183579 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.180432 2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 22:22:40.183579 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180489 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:22:40.183579 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180495 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:22:40.183579 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180499 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:22:40.183579 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180502 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:22:40.183579 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180505 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:22:40.183579 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180508 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:22:40.183579 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180511 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:22:40.183579 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180513 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:22:40.183579 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180516 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:22:40.183579 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180519 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:22:40.183579 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180521 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:22:40.184028 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180524 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:22:40.184028 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180527 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:22:40.184028 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180530 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:22:40.184028 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180532 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:22:40.184028 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180535 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:22:40.184028 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180537 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:22:40.184028 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180540 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:22:40.184028 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180544 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:22:40.184028 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180547 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:22:40.184028 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180550 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:22:40.184028 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180553 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:22:40.184028 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180556 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:22:40.184028 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180558 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:22:40.184028 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180561 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:22:40.184028 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180563 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:22:40.184028 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180566 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:22:40.184028 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180569 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:22:40.184028 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180571 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:22:40.184028 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180576 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:22:40.184473 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180578 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:22:40.184473 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180581 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:22:40.184473 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180583 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:22:40.184473 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180586 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:22:40.184473 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180588 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:22:40.184473 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180591 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:22:40.184473 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180593 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:22:40.184473 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180595 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:22:40.184473 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180598 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:22:40.184473 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180600 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:22:40.184473 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180603 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:22:40.184473 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180605 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:22:40.184473 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180608 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:22:40.184473 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180610 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:22:40.184473 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180612 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:22:40.184473 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180615 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:22:40.184473 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180618 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:22:40.184473 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180620 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:22:40.184473 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180622 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:22:40.184936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180625 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:22:40.184936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180627 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:22:40.184936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180630 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:22:40.184936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180633 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:22:40.184936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180635 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:22:40.184936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180638 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:22:40.184936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180641 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:22:40.184936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180643 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:22:40.184936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180646 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:22:40.184936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180648 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:22:40.184936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180651 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:22:40.184936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180654 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:22:40.184936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180657 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:22:40.184936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180660 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:22:40.184936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180663 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:22:40.184936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180666 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:22:40.184936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180668 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:22:40.184936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180670 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:22:40.184936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180673 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:22:40.184936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180675 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:22:40.185408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180677 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:22:40.185408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180680 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:22:40.185408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180683 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:22:40.185408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180685 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:22:40.185408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180687 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:22:40.185408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180690 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:22:40.185408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180692 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:22:40.185408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180695 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:22:40.185408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180697 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:22:40.185408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180700 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:22:40.185408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180703 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:22:40.185408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180706 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:22:40.185408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180708 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:22:40.185408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180711 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:22:40.185408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180713 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:22:40.185408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180716 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:22:40.185408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180718 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:22:40.185408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180720 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:22:40.185408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180724 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:22:40.185855 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180726 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:22:40.185855 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180729 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:22:40.185855 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180732 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:22:40.185855 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180735 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:22:40.185855 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180737 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:22:40.185855 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180740 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:22:40.185855 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180743 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:22:40.185855 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180746 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:22:40.185855 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180748 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:22:40.185855 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180751 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:22:40.185855 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180753 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:22:40.185855 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180757 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:22:40.185855 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180760 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:22:40.185855 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180763 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:22:40.185855 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180766 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:22:40.185855 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180768 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:22:40.185855 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180770 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:22:40.185855 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180773 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:22:40.186280 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.180801 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 22:22:40.186280 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180988 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:22:40.186280 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180994 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:22:40.186280 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180997 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:22:40.186280 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.180999 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:22:40.186280 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181002 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:22:40.186280 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181005 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:22:40.186280 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181007 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:22:40.186280 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181010 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:22:40.186280 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181012 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:22:40.186280 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181015 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:22:40.186280 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181018 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:22:40.186280 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181021 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:22:40.186280 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181023 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:22:40.186280 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181026 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:22:40.186280 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181028 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:22:40.186662 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181031 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:22:40.186662 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181033 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:22:40.186662 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181036 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:22:40.186662 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181039 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:22:40.186662 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181042 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:22:40.186662 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181044 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:22:40.186662 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181047 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:22:40.186662 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181050 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:22:40.186662 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181053 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:22:40.186662 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181056 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:22:40.186662 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181058 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:22:40.186662 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181061 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:22:40.186662 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181063 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:22:40.186662 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181066 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:22:40.186662 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181068 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:22:40.186662 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181071 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:22:40.186662 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181073 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:22:40.186662 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181075 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:22:40.186662 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181078 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:22:40.187133 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181080 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:22:40.187133 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181083 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:22:40.187133 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181085 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:22:40.187133 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181088 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:22:40.187133 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181090 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:22:40.187133 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181093 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:22:40.187133 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181095 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:22:40.187133 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181098 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:22:40.187133 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181100 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:22:40.187133 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181103 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:22:40.187133 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181105 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:22:40.187133 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181108 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:22:40.187133 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181111 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:22:40.187133 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181113 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:22:40.187133 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181116 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:22:40.187133 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181119 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:22:40.187133 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181122 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:22:40.187133 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181124 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:22:40.187133 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181127 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:22:40.187133 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181130 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:22:40.187601 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181138 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:22:40.187601 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181141 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:22:40.187601 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181143 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:22:40.187601 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181146 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:22:40.187601 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181148 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:22:40.187601 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181151 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:22:40.187601 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181153 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:22:40.187601 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181155 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:22:40.187601 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181158 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:22:40.187601 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181161 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:22:40.187601 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181163 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:22:40.187601 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181165 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:22:40.187601 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181168 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:22:40.187601 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181170 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:22:40.187601 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181173 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:22:40.187601 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181175 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:22:40.187601 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181177 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:22:40.187601 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181180 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:22:40.187601 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181182 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:22:40.187601 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181185 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:22:40.188093 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181187 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:22:40.188093 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181189 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:22:40.188093 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181193 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:22:40.188093 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181197 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:22:40.188093 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181200 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:22:40.188093 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181203 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:22:40.188093 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181206 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:22:40.188093 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181208 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:22:40.188093 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181211 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:22:40.188093 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181214 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:22:40.188093 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181216 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:22:40.188093 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181219 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:22:40.188093 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181221 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:22:40.188093 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181229 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:22:40.188093 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181232 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:22:40.188093 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181234 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:22:40.188093 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181237 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:22:40.188093 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181241 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:22:40.188093 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181245 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:22:40.188548 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181248 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:22:40.188548 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181251 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:22:40.188548 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181254 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:22:40.188548 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181256 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:22:40.188548 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181259 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:22:40.188548 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181261 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:22:40.188548 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181264 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:22:40.188548 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181266 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:22:40.188548 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181268 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:22:40.188548 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181271 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:22:40.188548 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181273 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:22:40.188548 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181276 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:22:40.188548 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181278 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:22:40.188548 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.181303 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 22:22:40.188548 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181344 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:22:40.188548 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181350 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:22:40.188936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181353 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:22:40.188936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181356 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:22:40.188936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181359 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:22:40.188936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181361 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:22:40.188936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181365 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:22:40.188936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181368 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:22:40.188936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181371 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:22:40.188936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181373 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:22:40.188936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181375 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:22:40.188936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181378 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:22:40.188936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181380 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:22:40.188936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181383 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:22:40.188936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181386 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:22:40.188936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181388 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:22:40.188936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181391 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:22:40.188936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181393 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:22:40.188936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181396 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:22:40.188936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181398 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:22:40.188936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181401 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:22:40.188936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181403 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:22:40.189426 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181405 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:22:40.189426 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181408 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:22:40.189426 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181410 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:22:40.189426 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181413 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:22:40.189426 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181415 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:22:40.189426 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181418 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:22:40.189426 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181420 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:22:40.189426 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181423 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:22:40.189426 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181425 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:22:40.189426 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181428 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:22:40.189426 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181430 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:22:40.189426 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181433 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:22:40.189426 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181435 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:22:40.189426 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181437 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:22:40.189426 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181440 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:22:40.189426 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181442 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:22:40.189426 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181445 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:22:40.189426 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181448 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:22:40.189426 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181450 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:22:40.189950 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181453 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:22:40.189950 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181456 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:22:40.189950 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181458 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:22:40.189950 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181461 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:22:40.189950 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181463 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:22:40.189950 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181466 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:22:40.189950 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181469 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:22:40.189950 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181472 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:22:40.189950 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181474 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:22:40.189950 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181476 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:22:40.189950 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181479 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:22:40.189950 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181481 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:22:40.189950 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181484 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:22:40.189950 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181486 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:22:40.189950 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181488 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:22:40.189950 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181491 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:22:40.189950 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181493 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:22:40.189950 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181496 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:22:40.189950 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181498 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:22:40.190408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181500 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:22:40.190408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181503 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:22:40.190408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181505 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:22:40.190408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181508 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:22:40.190408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181510 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:22:40.190408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181513 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:22:40.190408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181518 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:22:40.190408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181520 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:22:40.190408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181523 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:22:40.190408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181527 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:22:40.190408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181530 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:22:40.190408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181532 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:22:40.190408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181535 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:22:40.190408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181537 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:22:40.190408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181540 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:22:40.190408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181542 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:22:40.190408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181545 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:22:40.190408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181547 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:22:40.190408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181550 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:22:40.190408 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181553 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:22:40.190924 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181556 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:22:40.190924 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181559 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:22:40.190924 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181561 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:22:40.190924 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181564 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:22:40.190924 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181566 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:22:40.190924 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181568 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:22:40.190924 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181571 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:22:40.190924 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181573 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:22:40.190924 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181576 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:22:40.190924 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181578 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:22:40.190924 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181581 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:22:40.190924 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181585 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:22:40.190924 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181590 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:22:40.190924 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181593 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:22:40.190924 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181596 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:22:40.190924 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181599 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:22:40.190924 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181602 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:22:40.190924 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181605 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:22:40.190924 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181613 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:22:40.191385 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181616 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:22:40.191385 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181618 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:22:40.191385 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181622 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:22:40.191385 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181624 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:22:40.191385 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181627 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:22:40.191385 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181629 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:22:40.191385 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:40.181633 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:22:40.191385 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.181704 2576 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 22:22:40.191385 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.185583 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 01 22:22:40.191385 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.186652 2576 server.go:1014] "Starting client certificate rotation" Nov 01 22:22:40.191385 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.186775 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 22:22:40.191385 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.186812 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 22:22:40.191385 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.189031 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 22:22:40.191385 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.191269 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 22:22:40.203079 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.203051 2576 log.go:25] "Validated CRI v1 runtime API" Nov 01 22:22:40.208856 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.208823 2576 log.go:25] "Validated CRI v1 image API" Nov 01 22:22:40.209738 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.209711 2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 22:22:40.214528 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.214504 2576 fs.go:135] Filesystem UUIDs: map[259c1a59-854d-48cd-a40f-803aed05e1f1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e232985b-20c5-444c-8077-afd447b52533:/dev/nvme0n1p3] Nov 01 22:22:40.214622 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.214526 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 01 22:22:40.216482 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.216459 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 22:22:40.219162 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.219046 2576 manager.go:217] Machine: {Timestamp:2026-11-01 22:22:40.218547162 +0000 UTC m=+0.392356275 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099227 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec201604da364b01a05e96d1f942217b SystemUUID:ec201604-da36-4b01-a05e-96d1f942217b BootID:0b572456-2c6f-4dac-b385-51e3e47396b1 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d1:b0:d4:1d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d1:b0:d4:1d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:eb:6e:7e:40:99 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 22:22:40.219162 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.219161 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 22:22:40.219307 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.219295 2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 22:22:40.223021 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.222996 2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 22:22:40.223170 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.223024 2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-238.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 22:22:40.223220 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.223173 2576 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 22:22:40.223220 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.223181 2576 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 22:22:40.223220 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.223203 2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 22:22:40.223220 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.223218 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 22:22:40.224200 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.224189 2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 22:22:40.229382 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.229367 2576 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 22:22:40.229521 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.229510 2576 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 22:22:40.229551 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.229545 2576 kubelet.go:493] "Attempting to sync node with API server" Nov 01 22:22:40.229581 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.229555 2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 22:22:40.229581 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.229565 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 22:22:40.229581 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.229575 2576 kubelet.go:405] "Adding apiserver pod source" Nov 01 22:22:40.229656 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.229587 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 22:22:40.230059 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.230040 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 22:22:40.232370 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.232350 2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 22:22:40.232917 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.232899 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5jdqx" Nov 01 22:22:40.238042 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.238019 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5jdqx" Nov 01 22:22:40.239115 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.239094 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 22:22:40.240020 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.240001 2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 22:22:40.240102 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.240035 2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 22:22:40.240208 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.240194 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 22:22:40.240261 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.240211 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 22:22:40.240261 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.240218 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 22:22:40.240261 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.240226 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 22:22:40.240261 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.240232 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 22:22:40.240261 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.240239 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 22:22:40.240938 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.240864 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 22:22:40.240938 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.240874 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 22:22:40.240938 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.240882 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 22:22:40.240938 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.240889 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 22:22:40.240938 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.240897 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 22:22:40.240938 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.240907 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 22:22:40.241732 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.241720 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 22:22:40.241771 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.241735 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 22:22:40.245856 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.245823 2576 server.go:1263] "Started kubelet" Nov 01 22:22:40.245982 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.245945 2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 22:22:40.246045 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.245970 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 22:22:40.246093 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.246080 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 22:22:40.246674 ip-10-0-142-238 systemd[1]: Started Kubernetes Kubelet. Nov 01 22:22:40.248170 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.248149 2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 22:22:40.250964 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.250884 2576 server.go:317] "Adding debug handlers to kubelet server" Nov 01 22:22:40.252632 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.252586 2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:22:40.255401 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.255377 2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-142-238.us-east-2.compute.internal" not found Nov 01 22:22:40.255602 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.255583 2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:22:40.255914 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:40.255895 2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 01 22:22:40.257406 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.257070 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 22:22:40.258360 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.258344 2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 22:22:40.258451 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:40.258432 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-238.us-east-2.compute.internal\" not found" Nov 01 22:22:40.259156 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.259139 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 22:22:40.259156 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.259142 2576 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 22:22:40.259295 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.259180 2576 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 22:22:40.259295 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.259268 2576 reconstruct.go:97] "Volume reconstruction finished" Nov 01 22:22:40.259295 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.259280 2576 reconciler.go:29] "Reconciler: start to sync state" Nov 01 22:22:40.259722 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.259708 2576 factory.go:55] Registering systemd factory Nov 01 22:22:40.259784 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.259726 2576 factory.go:223] Registration of the systemd container factory successfully Nov 01 22:22:40.260025 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.260012 2576 factory.go:153] Registering CRI-O factory Nov 01 22:22:40.260071 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.260027 2576 factory.go:223] Registration of the crio container factory successfully Nov 01 22:22:40.260118 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.260081 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 22:22:40.260118 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.260102 2576 factory.go:103] Registering Raw factory Nov 01 22:22:40.260118 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.260113 2576 manager.go:1199] Started watching for new ooms in manager Nov 01 22:22:40.261221 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.261204 2576 manager.go:319] Starting recovery of all containers Nov 01 22:22:40.261892 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.261871 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:22:40.263436 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:40.263413 2576 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-238.us-east-2.compute.internal\" not found" node="ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:22:40.268439 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.268416 2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-142-238.us-east-2.compute.internal" not found Nov 01 22:22:40.271853 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.271817 2576 manager.go:324] Recovery completed Nov 01 22:22:40.276227 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.276209 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:22:40.278102 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.278085 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-238.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:22:40.278168 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.278120 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-238.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:22:40.278168 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.278132 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-238.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:22:40.279058 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.279037 2576 cpu_manager.go:226] "Starting" policy="none" Nov 01 22:22:40.279165 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.279077 2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 22:22:40.279165 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.279101 2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 22:22:40.279165 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.279129 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:22:40.280183 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.280163 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-238.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:22:40.280274 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.280201 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-238.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:22:40.280274 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.280221 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-238.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:22:40.281304 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.281290 2576 policy_none.go:50] "Start" Nov 01 22:22:40.281347 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.281308 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 22:22:40.281347 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.281319 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 22:22:40.283345 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.283330 2576 policy_none.go:44] "Start" Nov 01 22:22:40.312655 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.312451 2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 22:22:40.316699 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.316682 2576 manager.go:341] "Starting Device Plugin manager" Nov 01 22:22:40.316792 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:40.316720 2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 22:22:40.316792 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.316733 2576 server.go:87] "Starting device plugin registration server" Nov 01 22:22:40.317034 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.317020 2576 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 22:22:40.317099 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.317036 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 22:22:40.317147 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.317116 2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 22:22:40.317248 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.317234 2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 22:22:40.317248 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.317246 2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 22:22:40.317876 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:40.317850 2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 22:22:40.317954 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:40.317911 2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-238.us-east-2.compute.internal\" not found" Nov 01 22:22:40.330065 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.330047 2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-142-238.us-east-2.compute.internal" not found Nov 01 22:22:40.399099 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.399056 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 22:22:40.401252 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.400440 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 22:22:40.401252 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.400462 2576 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 22:22:40.401252 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.400485 2576 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 22:22:40.401252 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:40.400518 2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 01 22:22:40.403977 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.403957 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:22:40.417798 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.417744 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:22:40.418714 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.418698 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-238.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:22:40.418783 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.418730 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-238.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:22:40.418783 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.418746 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-238.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:22:40.418783 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.418771 2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:22:40.427765 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.427744 2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:22:40.501375 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.501336 2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-238.us-east-2.compute.internal"] Nov 01 22:22:40.503838 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.503805 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:22:40.505671 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.505654 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:22:40.545705 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.545687 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:22:40.548169 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.548153 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:22:40.558798 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.558783 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 22:22:40.561367 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.561351 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4f1e77a94f837e0f1b12b6a1e77eab89-config\") pod \"kube-apiserver-proxy-ip-10-0-142-238.us-east-2.compute.internal\" (UID: \"4f1e77a94f837e0f1b12b6a1e77eab89\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:22:40.561424 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.561376 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fbc5f96efa48930e21a66fef652c693e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal\" (UID: \"fbc5f96efa48930e21a66fef652c693e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:22:40.561424 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.561393 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbc5f96efa48930e21a66fef652c693e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal\" (UID: \"fbc5f96efa48930e21a66fef652c693e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:22:40.564341 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.564321 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 22:22:40.661649 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.661612 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fbc5f96efa48930e21a66fef652c693e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal\" (UID: \"fbc5f96efa48930e21a66fef652c693e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:22:40.661855 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.661655 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbc5f96efa48930e21a66fef652c693e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal\" (UID: \"fbc5f96efa48930e21a66fef652c693e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:22:40.661855 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.661681 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4f1e77a94f837e0f1b12b6a1e77eab89-config\") pod \"kube-apiserver-proxy-ip-10-0-142-238.us-east-2.compute.internal\" (UID: \"4f1e77a94f837e0f1b12b6a1e77eab89\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:22:40.661855 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.661730 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fbc5f96efa48930e21a66fef652c693e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal\" (UID: \"fbc5f96efa48930e21a66fef652c693e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:22:40.661855 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.661737 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4f1e77a94f837e0f1b12b6a1e77eab89-config\") pod \"kube-apiserver-proxy-ip-10-0-142-238.us-east-2.compute.internal\" (UID: \"4f1e77a94f837e0f1b12b6a1e77eab89\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:22:40.661855 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.661737 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbc5f96efa48930e21a66fef652c693e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal\" (UID: \"fbc5f96efa48930e21a66fef652c693e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:22:40.861662 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.861623 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:22:40.868375 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:40.868355 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:22:41.186444 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.186418 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 01 22:22:41.186999 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.186567 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 22:22:41.186999 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.186571 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 22:22:41.229798 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.229775 2576 apiserver.go:52] "Watching apiserver" Nov 01 22:22:41.238119 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.238099 2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 22:22:41.238394 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.238374 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr","openshift-dns/node-resolver-x4k8b","openshift-multus/multus-84vvx","openshift-multus/multus-additional-cni-plugins-jngxk","openshift-network-diagnostics/network-check-target-cfnmp","kube-system/konnectivity-agent-tmhm2","openshift-cluster-node-tuning-operator/tuned-mhvjj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal","kube-system/global-pull-secret-syncer-ttjsz","kube-system/kube-apiserver-proxy-ip-10-0-142-238.us-east-2.compute.internal","openshift-image-registry/node-ca-rkf2j","openshift-multus/network-metrics-daemon-xvgrl","openshift-network-operator/iptables-alerter-npsp2","openshift-ovn-kubernetes/ovnkube-node-6h899"] Nov 01 22:22:41.240710 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.240697 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.242640 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.242601 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 22:17:40 +0000 UTC" deadline="2028-07-29 12:02:02.329681554 +0000 UTC" Nov 01 22:22:41.242640 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.242639 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15253h39m21.087046677s" Nov 01 22:22:41.242752 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.242673 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x4k8b" Nov 01 22:22:41.245131 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.244427 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:22:41.245131 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.244522 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 22:22:41.245321 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.245305 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nz8z8\"" Nov 01 22:22:41.245914 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.245892 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rkf2j" Nov 01 22:22:41.246137 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.246120 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z74s5\"" Nov 01 22:22:41.246196 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.246142 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 22:22:41.246235 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.246227 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 22:22:41.248047 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.248029 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:22:41.250215 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.250196 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:22:41.250562 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.250549 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 22:22:41.252242 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.252225 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:22:41.252566 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:41.252288 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnmp" podUID="b439b405-8bca-4ce8-980d-a076cb2f11bf" Nov 01 22:22:41.254440 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.254424 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:22:41.254496 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:41.254481 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttjsz" podUID="fa603801-6c30-4cf1-ace8-30c8723e4a1e" Nov 01 22:22:41.256505 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.256493 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-npsp2" Nov 01 22:22:41.257988 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.257972 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 01 22:22:41.258710 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.258694 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tmhm2" Nov 01 22:22:41.261131 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.261112 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.263320 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.263301 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:22:41.263407 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:41.263354 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xvgrl" podUID="cc3de28d-1139-4677-81db-a114c8867933" Nov 01 22:22:41.264781 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.264762 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3af15ebe-c67a-40c5-bac0-24410542413b-cni-binary-copy\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:22:41.264864 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.264789 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-device-dir\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:22:41.264864 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.264807 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxhl8\" (UniqueName: \"kubernetes.io/projected/b439b405-8bca-4ce8-980d-a076cb2f11bf-kube-api-access-lxhl8\") pod \"network-check-target-cfnmp\" (UID: \"b439b405-8bca-4ce8-980d-a076cb2f11bf\") " pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:22:41.264864 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.264846 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-hostroot\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.264974 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.264890 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-modprobe-d\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.264974 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.264921 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3af15ebe-c67a-40c5-bac0-24410542413b-system-cni-dir\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:22:41.265031 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.264977 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3af15ebe-c67a-40c5-bac0-24410542413b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:22:41.265070 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.265053 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/682b0c5d-fa6f-4a17-8b72-a1f224cf5fec-serviceca\") pod \"node-ca-rkf2j\" (UID: \"682b0c5d-fa6f-4a17-8b72-a1f224cf5fec\") " pod="openshift-image-registry/node-ca-rkf2j" Nov 01 22:22:41.265118 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.265097 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/179a164c-f6dd-42f9-9428-f241f81afff5-multus-daemon-config\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.265150 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.265136 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fa603801-6c30-4cf1-ace8-30c8723e4a1e-kubelet-config\") pod \"global-pull-secret-syncer-ttjsz\" (UID: \"fa603801-6c30-4cf1-ace8-30c8723e4a1e\") " pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:22:41.265181 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.265158 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-kubernetes\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.265215 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.265196 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-sys-fs\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:22:41.265245 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.265221 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-system-cni-dir\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.265245 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.265240 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-host\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.265302 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.265255 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f71460db-658f-45ed-980f-da65509e2bc5-hosts-file\") pod \"node-resolver-x4k8b\" (UID: \"f71460db-658f-45ed-980f-da65509e2bc5\") " pod="openshift-dns/node-resolver-x4k8b" Nov 01 22:22:41.265302 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.265275 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3bed4cac-d251-49be-9eb6-ce5cdb6852de-agent-certs\") pod \"konnectivity-agent-tmhm2\" (UID: \"3bed4cac-d251-49be-9eb6-ce5cdb6852de\") " pod="kube-system/konnectivity-agent-tmhm2" Nov 01 22:22:41.265302 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.265289 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-var-lib-kubelet\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.265391 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.265326 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sstcj\" (UniqueName: \"kubernetes.io/projected/179a164c-f6dd-42f9-9428-f241f81afff5-kube-api-access-sstcj\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.265391 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.265350 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8-host-slash\") pod \"iptables-alerter-npsp2\" (UID: \"3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8\") " pod="openshift-network-operator/iptables-alerter-npsp2" Nov 01 22:22:41.265391 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.265374 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-var-lib-kubelet\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.265566 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.265423 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-sysconfig\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.265566 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.265438 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-run\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.265566 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.265477 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkg5t\" (UniqueName: \"kubernetes.io/projected/3af15ebe-c67a-40c5-bac0-24410542413b-kube-api-access-nkg5t\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:22:41.265566 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.265505 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/682b0c5d-fa6f-4a17-8b72-a1f224cf5fec-host\") pod \"node-ca-rkf2j\" (UID: \"682b0c5d-fa6f-4a17-8b72-a1f224cf5fec\") " pod="openshift-image-registry/node-ca-rkf2j" Nov 01 22:22:41.265566 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.265527 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/179a164c-f6dd-42f9-9428-f241f81afff5-cni-binary-copy\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.265566 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.265542 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-multus-socket-dir-parent\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.265865 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.265606 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c959caa-7239-4b81-8c19-c867fc6c48e6-tmp\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.265865 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.265641 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3af15ebe-c67a-40c5-bac0-24410542413b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:22:41.265865 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.265672 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa603801-6c30-4cf1-ace8-30c8723e4a1e-original-pull-secret\") pod \"global-pull-secret-syncer-ttjsz\" (UID: \"fa603801-6c30-4cf1-ace8-30c8723e4a1e\") " pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:22:41.265865 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.265673 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.265865 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.265721 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-socket-dir\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:22:41.265865 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.265762 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-sysctl-d\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.265865 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.265786 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3af15ebe-c67a-40c5-bac0-24410542413b-cnibin\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:22:41.265865 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.265801 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqg4h\" (UniqueName: \"kubernetes.io/projected/3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8-kube-api-access-mqg4h\") pod \"iptables-alerter-npsp2\" (UID: \"3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8\") " pod="openshift-network-operator/iptables-alerter-npsp2" Nov 01 22:22:41.265865 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.265850 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9q9h\" (UniqueName: \"kubernetes.io/projected/14cdd491-7383-4cc5-ac4f-60a75110d764-kube-api-access-n9q9h\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:22:41.266295 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.265877 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-etc-kubernetes\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.266295 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.265904 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-var-lib-cni-bin\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.266295 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.265936 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-systemd\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.266295 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.265970 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-registration-dir\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:22:41.266295 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.266004 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-run-k8s-cni-cncf-io\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.266295 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.266048 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s67jv\" (UniqueName: \"kubernetes.io/projected/5c959caa-7239-4b81-8c19-c867fc6c48e6-kube-api-access-s67jv\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.266295 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.266072 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3bed4cac-d251-49be-9eb6-ce5cdb6852de-konnectivity-ca\") pod \"konnectivity-agent-tmhm2\" (UID: \"3bed4cac-d251-49be-9eb6-ce5cdb6852de\") " pod="kube-system/konnectivity-agent-tmhm2" Nov 01 22:22:41.266295 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.266110 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-tuned\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.266295 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.266138 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-run-netns\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.266295 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.266175 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-var-lib-cni-multus\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.266295 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.266208 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-sysctl-conf\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.266295 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.266240 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-lib-modules\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.266295 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.266266 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzs82\" (UniqueName: \"kubernetes.io/projected/f71460db-658f-45ed-980f-da65509e2bc5-kube-api-access-pzs82\") pod \"node-resolver-x4k8b\" (UID: \"f71460db-658f-45ed-980f-da65509e2bc5\") " pod="openshift-dns/node-resolver-x4k8b" Nov 01 22:22:41.266295 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.266296 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fa603801-6c30-4cf1-ace8-30c8723e4a1e-dbus\") pod \"global-pull-secret-syncer-ttjsz\" (UID: \"fa603801-6c30-4cf1-ace8-30c8723e4a1e\") " pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:22:41.266864 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.266319 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd2dd\" (UniqueName: \"kubernetes.io/projected/682b0c5d-fa6f-4a17-8b72-a1f224cf5fec-kube-api-access-fd2dd\") pod \"node-ca-rkf2j\" (UID: \"682b0c5d-fa6f-4a17-8b72-a1f224cf5fec\") " pod="openshift-image-registry/node-ca-rkf2j" Nov 01 22:22:41.266864 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.266342 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-sys\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.266864 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.266365 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f71460db-658f-45ed-980f-da65509e2bc5-tmp-dir\") pod \"node-resolver-x4k8b\" (UID: \"f71460db-658f-45ed-980f-da65509e2bc5\") " pod="openshift-dns/node-resolver-x4k8b" Nov 01 22:22:41.266864 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.266401 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-multus-conf-dir\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.266864 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.266437 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-run-multus-certs\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.266864 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.266490 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:22:41.266864 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.266514 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-etc-selinux\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:22:41.266864 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.266538 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-multus-cni-dir\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.266864 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.266565 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3af15ebe-c67a-40c5-bac0-24410542413b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:22:41.266864 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.266598 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8-iptables-alerter-script\") pod \"iptables-alerter-npsp2\" (UID: \"3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8\") " pod="openshift-network-operator/iptables-alerter-npsp2" Nov 01 22:22:41.266864 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.266625 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-cnibin\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.275005 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.274987 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 22:22:41.290352 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.290332 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cphdx" Nov 01 22:22:41.297033 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.297016 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cphdx" Nov 01 22:22:41.345138 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.345111 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 22:22:41.345389 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.345151 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 22:22:41.345458 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.345365 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:22:41.345458 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.345369 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 22:22:41.345529 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.345511 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 22:22:41.345529 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.345518 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 22:22:41.346778 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.346705 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 22:22:41.346778 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.346731 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 22:22:41.346778 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.346739 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 22:22:41.346778 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.346749 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 22:22:41.346778 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.346731 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 22:22:41.346778 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.346739 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xdns4\"" Nov 01 22:22:41.347011 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.346793 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 22:22:41.347011 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.346863 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 22:22:41.347011 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.346904 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5sq9j\"" Nov 01 22:22:41.347011 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.346923 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 22:22:41.348023 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.348006 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qzblw\"" Nov 01 22:22:41.348125 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.348018 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 22:22:41.348412 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.348373 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jvq4g\"" Nov 01 22:22:41.348412 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.348396 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 22:22:41.348412 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.348398 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6vwc2\"" Nov 01 22:22:41.348569 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.348442 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 22:22:41.348569 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.348446 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 22:22:41.348569 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.348471 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 22:22:41.348569 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.348488 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 22:22:41.348569 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.348528 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 22:22:41.348722 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.348577 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-th659\"" Nov 01 22:22:41.349803 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.349761 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9gzz5\"" Nov 01 22:22:41.349943 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.349928 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 22:22:41.361618 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.360177 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 22:22:41.367798 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.367774 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-socket-dir\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:22:41.367919 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.367817 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.367919 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.367863 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-sysctl-d\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.367919 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.367891 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3af15ebe-c67a-40c5-bac0-24410542413b-cnibin\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:22:41.368072 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.367921 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqg4h\" (UniqueName: \"kubernetes.io/projected/3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8-kube-api-access-mqg4h\") pod \"iptables-alerter-npsp2\" (UID: \"3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8\") " pod="openshift-network-operator/iptables-alerter-npsp2" Nov 01 22:22:41.368072 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.367976 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9q9h\" (UniqueName: \"kubernetes.io/projected/14cdd491-7383-4cc5-ac4f-60a75110d764-kube-api-access-n9q9h\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:22:41.368072 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.367986 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3af15ebe-c67a-40c5-bac0-24410542413b-cnibin\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:22:41.368072 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.368038 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-etc-kubernetes\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.368072 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.368001 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-etc-kubernetes\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.368072 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.368057 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-sysctl-d\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.368332 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.368067 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/45d5cfa6-bf31-4fad-8003-d28fe9a37183-ovnkube-config\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.368332 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.368101 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-var-lib-cni-bin\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.368332 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.368117 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-systemd\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.368332 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.368134 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-registration-dir\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:22:41.368332 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.367989 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-socket-dir\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:22:41.368332 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.368152 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-run-k8s-cni-cncf-io\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.368332 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.368196 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-systemd\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.368332 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.368198 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s67jv\" (UniqueName: \"kubernetes.io/projected/5c959caa-7239-4b81-8c19-c867fc6c48e6-kube-api-access-s67jv\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.368332 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.368218 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-run-k8s-cni-cncf-io\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.368332 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.368245 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3bed4cac-d251-49be-9eb6-ce5cdb6852de-konnectivity-ca\") pod \"konnectivity-agent-tmhm2\" (UID: \"3bed4cac-d251-49be-9eb6-ce5cdb6852de\") " pod="kube-system/konnectivity-agent-tmhm2" Nov 01 22:22:41.368332 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.368242 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-var-lib-cni-bin\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.368332 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.368269 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-registration-dir\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:22:41.369888 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.368347 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/45d5cfa6-bf31-4fad-8003-d28fe9a37183-ovnkube-script-lib\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.369888 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.368434 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-tuned\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.369888 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.368458 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-run-netns\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.369888 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.368504 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-run-netns\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.369888 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.368533 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-var-lib-cni-multus\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.369888 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.368568 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-kubelet\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.369888 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.368602 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-slash\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.369888 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.368620 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-var-lib-cni-multus\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.369888 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.368671 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-etc-openvswitch\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.369888 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.368696 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-run-openvswitch\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.369888 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.368717 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-cni-netd\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.369888 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.368745 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-sysctl-conf\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.369888 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.368765 2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 22:22:41.369888 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.368781 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-lib-modules\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.369888 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.368892 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzs82\" (UniqueName: \"kubernetes.io/projected/f71460db-658f-45ed-980f-da65509e2bc5-kube-api-access-pzs82\") pod \"node-resolver-x4k8b\" (UID: \"f71460db-658f-45ed-980f-da65509e2bc5\") " pod="openshift-dns/node-resolver-x4k8b" Nov 01 22:22:41.369888 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.368918 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-sysctl-conf\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.369888 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.368922 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fa603801-6c30-4cf1-ace8-30c8723e4a1e-dbus\") pod \"global-pull-secret-syncer-ttjsz\" (UID: \"fa603801-6c30-4cf1-ace8-30c8723e4a1e\") " pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:22:41.369888 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.368896 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3bed4cac-d251-49be-9eb6-ce5cdb6852de-konnectivity-ca\") pod \"konnectivity-agent-tmhm2\" (UID: \"3bed4cac-d251-49be-9eb6-ce5cdb6852de\") " pod="kube-system/konnectivity-agent-tmhm2" Nov 01 22:22:41.370709 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.369004 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd2dd\" (UniqueName: \"kubernetes.io/projected/682b0c5d-fa6f-4a17-8b72-a1f224cf5fec-kube-api-access-fd2dd\") pod \"node-ca-rkf2j\" (UID: \"682b0c5d-fa6f-4a17-8b72-a1f224cf5fec\") " pod="openshift-image-registry/node-ca-rkf2j" Nov 01 22:22:41.370709 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.369032 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-var-lib-openvswitch\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.370709 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.369066 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-sys\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.370709 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.369077 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-lib-modules\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.370709 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.369087 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f71460db-658f-45ed-980f-da65509e2bc5-tmp-dir\") pod \"node-resolver-x4k8b\" (UID: \"f71460db-658f-45ed-980f-da65509e2bc5\") " pod="openshift-dns/node-resolver-x4k8b" Nov 01 22:22:41.370709 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.369112 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-multus-conf-dir\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.370709 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.369114 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fa603801-6c30-4cf1-ace8-30c8723e4a1e-dbus\") pod \"global-pull-secret-syncer-ttjsz\" (UID: \"fa603801-6c30-4cf1-ace8-30c8723e4a1e\") " pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:22:41.370709 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.369264 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-sys\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.370709 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.369269 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-run-multus-certs\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.370709 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.369306 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-run-multus-certs\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.370709 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.369311 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:22:41.370709 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.369345 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-etc-selinux\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:22:41.370709 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.369351 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-multus-conf-dir\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.370709 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.369370 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-multus-cni-dir\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.370709 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.369425 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc3de28d-1139-4677-81db-a114c8867933-metrics-certs\") pod \"network-metrics-daemon-xvgrl\" (UID: \"cc3de28d-1139-4677-81db-a114c8867933\") " pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:22:41.370709 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.369439 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:22:41.370709 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.369449 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-etc-selinux\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:22:41.371362 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.369469 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3af15ebe-c67a-40c5-bac0-24410542413b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:22:41.371362 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.369530 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-multus-cni-dir\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.371362 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.369563 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8-iptables-alerter-script\") pod \"iptables-alerter-npsp2\" (UID: \"3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8\") " pod="openshift-network-operator/iptables-alerter-npsp2" Nov 01 22:22:41.371362 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.369602 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f71460db-658f-45ed-980f-da65509e2bc5-tmp-dir\") pod \"node-resolver-x4k8b\" (UID: \"f71460db-658f-45ed-980f-da65509e2bc5\") " pod="openshift-dns/node-resolver-x4k8b" Nov 01 22:22:41.371362 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.369624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-cnibin\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.371362 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.369670 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3af15ebe-c67a-40c5-bac0-24410542413b-cni-binary-copy\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:22:41.371362 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.369680 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-cnibin\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.371362 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.369709 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-device-dir\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:22:41.371362 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.369735 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxhl8\" (UniqueName: \"kubernetes.io/projected/b439b405-8bca-4ce8-980d-a076cb2f11bf-kube-api-access-lxhl8\") pod \"network-check-target-cfnmp\" (UID: \"b439b405-8bca-4ce8-980d-a076cb2f11bf\") " pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:22:41.371362 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.369757 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-hostroot\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.371362 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.369789 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-hostroot\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.371362 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.369807 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-device-dir\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:22:41.371362 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.369821 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-modprobe-d\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.371362 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.370050 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3af15ebe-c67a-40c5-bac0-24410542413b-system-cni-dir\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:22:41.371362 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.370099 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-modprobe-d\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.371362 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.370148 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3af15ebe-c67a-40c5-bac0-24410542413b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:22:41.371362 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.370168 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3af15ebe-c67a-40c5-bac0-24410542413b-system-cni-dir\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:22:41.372002 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.370221 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/682b0c5d-fa6f-4a17-8b72-a1f224cf5fec-serviceca\") pod \"node-ca-rkf2j\" (UID: \"682b0c5d-fa6f-4a17-8b72-a1f224cf5fec\") " pod="openshift-image-registry/node-ca-rkf2j" Nov 01 22:22:41.372002 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.370250 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/179a164c-f6dd-42f9-9428-f241f81afff5-multus-daemon-config\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.372002 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.370367 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-run-systemd\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.372002 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.370392 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-node-log\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.372002 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.370421 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-cni-bin\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.372002 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.370458 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fa603801-6c30-4cf1-ace8-30c8723e4a1e-kubelet-config\") pod \"global-pull-secret-syncer-ttjsz\" (UID: \"fa603801-6c30-4cf1-ace8-30c8723e4a1e\") " pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:22:41.372002 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.370523 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fa603801-6c30-4cf1-ace8-30c8723e4a1e-kubelet-config\") pod \"global-pull-secret-syncer-ttjsz\" (UID: \"fa603801-6c30-4cf1-ace8-30c8723e4a1e\") " pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:22:41.372002 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.370539 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3af15ebe-c67a-40c5-bac0-24410542413b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:22:41.372002 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.370649 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8-iptables-alerter-script\") pod \"iptables-alerter-npsp2\" (UID: \"3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8\") " pod="openshift-network-operator/iptables-alerter-npsp2" Nov 01 22:22:41.372002 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.370705 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-kubernetes\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.372002 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.370731 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-sys-fs\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:22:41.372002 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.370757 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-kubernetes\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.372002 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.370753 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-system-cni-dir\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.372002 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.370803 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-systemd-units\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.372002 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.370815 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-sys-fs\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:22:41.372002 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.370846 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-log-socket\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.372002 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.370864 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/682b0c5d-fa6f-4a17-8b72-a1f224cf5fec-serviceca\") pod \"node-ca-rkf2j\" (UID: \"682b0c5d-fa6f-4a17-8b72-a1f224cf5fec\") " pod="openshift-image-registry/node-ca-rkf2j" Nov 01 22:22:41.372607 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.370870 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-host\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.372607 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.370887 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-system-cni-dir\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.372607 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.370910 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f71460db-658f-45ed-980f-da65509e2bc5-hosts-file\") pod \"node-resolver-x4k8b\" (UID: \"f71460db-658f-45ed-980f-da65509e2bc5\") " pod="openshift-dns/node-resolver-x4k8b" Nov 01 22:22:41.372607 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.370943 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-host\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.372607 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.370959 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f71460db-658f-45ed-980f-da65509e2bc5-hosts-file\") pod \"node-resolver-x4k8b\" (UID: \"f71460db-658f-45ed-980f-da65509e2bc5\") " pod="openshift-dns/node-resolver-x4k8b" Nov 01 22:22:41.372607 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.370976 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3bed4cac-d251-49be-9eb6-ce5cdb6852de-agent-certs\") pod \"konnectivity-agent-tmhm2\" (UID: \"3bed4cac-d251-49be-9eb6-ce5cdb6852de\") " pod="kube-system/konnectivity-agent-tmhm2" Nov 01 22:22:41.372607 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.371020 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-var-lib-kubelet\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.372607 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.371045 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sstcj\" (UniqueName: \"kubernetes.io/projected/179a164c-f6dd-42f9-9428-f241f81afff5-kube-api-access-sstcj\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.372607 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.371080 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nvbs\" (UniqueName: \"kubernetes.io/projected/cc3de28d-1139-4677-81db-a114c8867933-kube-api-access-6nvbs\") pod \"network-metrics-daemon-xvgrl\" (UID: \"cc3de28d-1139-4677-81db-a114c8867933\") " pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:22:41.372607 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.371103 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/45d5cfa6-bf31-4fad-8003-d28fe9a37183-env-overrides\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.372607 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.371131 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-var-lib-kubelet\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.372607 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.371125 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-run-netns\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.372607 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.371189 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/45d5cfa6-bf31-4fad-8003-d28fe9a37183-ovn-node-metrics-cert\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.372607 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.371235 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw4f7\" (UniqueName: \"kubernetes.io/projected/45d5cfa6-bf31-4fad-8003-d28fe9a37183-kube-api-access-kw4f7\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.372607 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.371235 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/179a164c-f6dd-42f9-9428-f241f81afff5-multus-daemon-config\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.372607 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.371280 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8-host-slash\") pod \"iptables-alerter-npsp2\" (UID: \"3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8\") " pod="openshift-network-operator/iptables-alerter-npsp2" Nov 01 22:22:41.372607 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.371327 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-run-ovn\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.373240 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.371366 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-run-ovn-kubernetes\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.373240 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.371395 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-var-lib-kubelet\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.373240 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.371437 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8-host-slash\") pod \"iptables-alerter-npsp2\" (UID: \"3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8\") " pod="openshift-network-operator/iptables-alerter-npsp2" Nov 01 22:22:41.373240 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.371482 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-var-lib-kubelet\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.373240 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.371513 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-sysconfig\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.373240 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.371554 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-run\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.373240 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.371590 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkg5t\" (UniqueName: \"kubernetes.io/projected/3af15ebe-c67a-40c5-bac0-24410542413b-kube-api-access-nkg5t\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:22:41.373240 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.371646 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/682b0c5d-fa6f-4a17-8b72-a1f224cf5fec-host\") pod \"node-ca-rkf2j\" (UID: \"682b0c5d-fa6f-4a17-8b72-a1f224cf5fec\") " pod="openshift-image-registry/node-ca-rkf2j" Nov 01 22:22:41.373240 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.371652 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3af15ebe-c67a-40c5-bac0-24410542413b-cni-binary-copy\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:22:41.373240 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.371670 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/179a164c-f6dd-42f9-9428-f241f81afff5-cni-binary-copy\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.373240 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.371718 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/682b0c5d-fa6f-4a17-8b72-a1f224cf5fec-host\") pod \"node-ca-rkf2j\" (UID: \"682b0c5d-fa6f-4a17-8b72-a1f224cf5fec\") " pod="openshift-image-registry/node-ca-rkf2j" Nov 01 22:22:41.373240 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.371765 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-multus-socket-dir-parent\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.373240 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.371781 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-run\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.373240 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.371819 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c959caa-7239-4b81-8c19-c867fc6c48e6-tmp\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.373240 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.371879 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3af15ebe-c67a-40c5-bac0-24410542413b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:22:41.373240 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.371943 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa603801-6c30-4cf1-ace8-30c8723e4a1e-original-pull-secret\") pod \"global-pull-secret-syncer-ttjsz\" (UID: \"fa603801-6c30-4cf1-ace8-30c8723e4a1e\") " pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:22:41.373240 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.371993 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-sysconfig\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.373240 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.372149 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/179a164c-f6dd-42f9-9428-f241f81afff5-cni-binary-copy\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.373982 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:41.372551 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:41.373982 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.372555 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-multus-socket-dir-parent\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.373982 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.372608 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3af15ebe-c67a-40c5-bac0-24410542413b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:22:41.373982 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:41.372638 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa603801-6c30-4cf1-ace8-30c8723e4a1e-original-pull-secret podName:fa603801-6c30-4cf1-ace8-30c8723e4a1e nodeName:}" failed. No retries permitted until 2026-11-01 22:22:41.872617467 +0000 UTC m=+2.046426591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa603801-6c30-4cf1-ace8-30c8723e4a1e-original-pull-secret") pod "global-pull-secret-syncer-ttjsz" (UID: "fa603801-6c30-4cf1-ace8-30c8723e4a1e") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:41.373982 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.373448 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3af15ebe-c67a-40c5-bac0-24410542413b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:22:41.374792 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.374768 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3bed4cac-d251-49be-9eb6-ce5cdb6852de-agent-certs\") pod \"konnectivity-agent-tmhm2\" (UID: \"3bed4cac-d251-49be-9eb6-ce5cdb6852de\") " pod="kube-system/konnectivity-agent-tmhm2" Nov 01 22:22:41.376209 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.375817 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c959caa-7239-4b81-8c19-c867fc6c48e6-tmp\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.376209 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.376031 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-tuned\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.377290 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.377091 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tmhm2" Nov 01 22:22:41.382450 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:41.382428 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:22:41.382450 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:41.382455 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:22:41.382609 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:41.382467 2576 projected.go:196] Error preparing data for projected volume kube-api-access-lxhl8 for pod openshift-network-diagnostics/network-check-target-cfnmp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:41.382609 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:41.382530 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b439b405-8bca-4ce8-980d-a076cb2f11bf-kube-api-access-lxhl8 podName:b439b405-8bca-4ce8-980d-a076cb2f11bf nodeName:}" failed. No retries permitted until 2026-11-01 22:22:41.88251084 +0000 UTC m=+2.056319962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lxhl8" (UniqueName: "kubernetes.io/projected/b439b405-8bca-4ce8-980d-a076cb2f11bf-kube-api-access-lxhl8") pod "network-check-target-cfnmp" (UID: "b439b405-8bca-4ce8-980d-a076cb2f11bf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:41.384651 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.384629 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzs82\" (UniqueName: \"kubernetes.io/projected/f71460db-658f-45ed-980f-da65509e2bc5-kube-api-access-pzs82\") pod \"node-resolver-x4k8b\" (UID: \"f71460db-658f-45ed-980f-da65509e2bc5\") " pod="openshift-dns/node-resolver-x4k8b" Nov 01 22:22:41.385668 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.385181 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9q9h\" (UniqueName: \"kubernetes.io/projected/14cdd491-7383-4cc5-ac4f-60a75110d764-kube-api-access-n9q9h\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:22:41.385668 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.385224 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sstcj\" (UniqueName: \"kubernetes.io/projected/179a164c-f6dd-42f9-9428-f241f81afff5-kube-api-access-sstcj\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.385668 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.385631 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkg5t\" (UniqueName: \"kubernetes.io/projected/3af15ebe-c67a-40c5-bac0-24410542413b-kube-api-access-nkg5t\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:22:41.385891 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.385759 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd2dd\" (UniqueName: \"kubernetes.io/projected/682b0c5d-fa6f-4a17-8b72-a1f224cf5fec-kube-api-access-fd2dd\") pod \"node-ca-rkf2j\" (UID: \"682b0c5d-fa6f-4a17-8b72-a1f224cf5fec\") " pod="openshift-image-registry/node-ca-rkf2j" Nov 01 22:22:41.386516 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.386495 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqg4h\" (UniqueName: \"kubernetes.io/projected/3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8-kube-api-access-mqg4h\") pod \"iptables-alerter-npsp2\" (UID: \"3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8\") " pod="openshift-network-operator/iptables-alerter-npsp2" Nov 01 22:22:41.388057 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.388036 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s67jv\" (UniqueName: \"kubernetes.io/projected/5c959caa-7239-4b81-8c19-c867fc6c48e6-kube-api-access-s67jv\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.398005 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.397982 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-84vvx" Nov 01 22:22:41.472755 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.472671 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.472755 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.472716 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/45d5cfa6-bf31-4fad-8003-d28fe9a37183-ovnkube-config\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.472755 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.472750 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/45d5cfa6-bf31-4fad-8003-d28fe9a37183-ovnkube-script-lib\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.473055 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.472777 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-kubelet\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.473055 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.472788 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.473055 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.472823 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-kubelet\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.473055 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.472872 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-slash\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.473055 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.472897 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-etc-openvswitch\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.473055 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.472920 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-run-openvswitch\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.473055 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.472942 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-cni-netd\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.473055 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.472970 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-var-lib-openvswitch\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.473055 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.473005 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc3de28d-1139-4677-81db-a114c8867933-metrics-certs\") pod \"network-metrics-daemon-xvgrl\" (UID: \"cc3de28d-1139-4677-81db-a114c8867933\") " pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:22:41.473501 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:41.473087 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:41.473501 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.473101 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-run-systemd\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.473501 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.473124 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-run-openvswitch\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.473501 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.473134 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-node-log\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.473501 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:41.473149 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc3de28d-1139-4677-81db-a114c8867933-metrics-certs podName:cc3de28d-1139-4677-81db-a114c8867933 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:41.973130863 +0000 UTC m=+2.146939979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc3de28d-1139-4677-81db-a114c8867933-metrics-certs") pod "network-metrics-daemon-xvgrl" (UID: "cc3de28d-1139-4677-81db-a114c8867933") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:41.473501 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.473165 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-slash\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.473501 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.473174 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-node-log\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.473501 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.473188 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-cni-netd\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.473501 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.473209 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-etc-openvswitch\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.473501 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.473225 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-var-lib-openvswitch\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.473501 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.473261 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-run-systemd\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.473501 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.473273 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-cni-bin\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.473501 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.473301 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-systemd-units\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.473501 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.473322 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-log-socket\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.473501 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.473392 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nvbs\" (UniqueName: \"kubernetes.io/projected/cc3de28d-1139-4677-81db-a114c8867933-kube-api-access-6nvbs\") pod \"network-metrics-daemon-xvgrl\" (UID: \"cc3de28d-1139-4677-81db-a114c8867933\") " pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:22:41.473501 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.473424 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/45d5cfa6-bf31-4fad-8003-d28fe9a37183-env-overrides\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.473501 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.473441 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/45d5cfa6-bf31-4fad-8003-d28fe9a37183-ovnkube-script-lib\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.474414 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.473460 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/45d5cfa6-bf31-4fad-8003-d28fe9a37183-ovnkube-config\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.474414 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.473449 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-run-netns\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.474414 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.473504 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-cni-bin\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.474414 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.473516 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/45d5cfa6-bf31-4fad-8003-d28fe9a37183-ovn-node-metrics-cert\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.474414 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.473540 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-run-netns\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.474414 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.473546 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw4f7\" (UniqueName: \"kubernetes.io/projected/45d5cfa6-bf31-4fad-8003-d28fe9a37183-kube-api-access-kw4f7\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.474414 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.473612 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-systemd-units\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.474414 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.473659 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-log-socket\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.474414 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.473784 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/45d5cfa6-bf31-4fad-8003-d28fe9a37183-env-overrides\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.474414 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.473809 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-run-ovn\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.474414 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.473866 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-run-ovn\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.474414 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.473874 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-run-ovn-kubernetes\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.474414 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.473907 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-run-ovn-kubernetes\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.475841 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.475813 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/45d5cfa6-bf31-4fad-8003-d28fe9a37183-ovn-node-metrics-cert\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.485723 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:41.485697 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f1e77a94f837e0f1b12b6a1e77eab89.slice/crio-caa1cde0c045e60f020bda4fa2b9c477955b5b0f1ed5dda0dc33c7b8df8916b6 WatchSource:0}: Error finding container caa1cde0c045e60f020bda4fa2b9c477955b5b0f1ed5dda0dc33c7b8df8916b6: Status 404 returned error can't find the container with id caa1cde0c045e60f020bda4fa2b9c477955b5b0f1ed5dda0dc33c7b8df8916b6 Nov 01 22:22:41.486377 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:41.486360 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc5f96efa48930e21a66fef652c693e.slice/crio-1d8b4c5a4a30cc4854e7d670f40126715432e1fba59c5f74cf836857393e1784 WatchSource:0}: Error finding container 1d8b4c5a4a30cc4854e7d670f40126715432e1fba59c5f74cf836857393e1784: Status 404 returned error can't find the container with id 1d8b4c5a4a30cc4854e7d670f40126715432e1fba59c5f74cf836857393e1784 Nov 01 22:22:41.486971 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.486955 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nvbs\" (UniqueName: \"kubernetes.io/projected/cc3de28d-1139-4677-81db-a114c8867933-kube-api-access-6nvbs\") pod \"network-metrics-daemon-xvgrl\" (UID: \"cc3de28d-1139-4677-81db-a114c8867933\") " pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:22:41.487240 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.487223 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw4f7\" (UniqueName: \"kubernetes.io/projected/45d5cfa6-bf31-4fad-8003-d28fe9a37183-kube-api-access-kw4f7\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.490245 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.490227 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:22:41.572632 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.572602 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:22:41.582869 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:41.582847 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c959caa_7239_4b81_8c19_c867fc6c48e6.slice/crio-a2d0ec52d05829919a9fe2e609c56582be559d0438c6ddc983191adb7751a8ce WatchSource:0}: Error finding container a2d0ec52d05829919a9fe2e609c56582be559d0438c6ddc983191adb7751a8ce: Status 404 returned error can't find the container with id a2d0ec52d05829919a9fe2e609c56582be559d0438c6ddc983191adb7751a8ce Nov 01 22:22:41.597365 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.597329 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x4k8b" Nov 01 22:22:41.603581 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.603566 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rkf2j" Nov 01 22:22:41.606291 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:41.606262 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf71460db_658f_45ed_980f_da65509e2bc5.slice/crio-abacbe9c358eb7c91c09dbba58a8c965c778bac8677000afff60f2a0ecc3275b WatchSource:0}: Error finding container abacbe9c358eb7c91c09dbba58a8c965c778bac8677000afff60f2a0ecc3275b: Status 404 returned error can't find the container with id abacbe9c358eb7c91c09dbba58a8c965c778bac8677000afff60f2a0ecc3275b Nov 01 22:22:41.612337 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:41.612311 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod682b0c5d_fa6f_4a17_8b72_a1f224cf5fec.slice/crio-0dc901321c676181e22a19ae8becbeb76d04ef9b84ee2856e6a59c9ffffbbae2 WatchSource:0}: Error finding container 0dc901321c676181e22a19ae8becbeb76d04ef9b84ee2856e6a59c9ffffbbae2: Status 404 returned error can't find the container with id 0dc901321c676181e22a19ae8becbeb76d04ef9b84ee2856e6a59c9ffffbbae2 Nov 01 22:22:41.644501 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.644476 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:22:41.654827 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:41.654800 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af15ebe_c67a_40c5_bac0_24410542413b.slice/crio-2f3d84313e3ca3144e9101369e8768171707b39c1a5069cface47f369f261a6e WatchSource:0}: Error finding container 2f3d84313e3ca3144e9101369e8768171707b39c1a5069cface47f369f261a6e: Status 404 returned error can't find the container with id 2f3d84313e3ca3144e9101369e8768171707b39c1a5069cface47f369f261a6e Nov 01 22:22:41.658329 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.658311 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:22:41.662076 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.662054 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-npsp2" Nov 01 22:22:41.670022 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:41.669995 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14cdd491_7383_4cc5_ac4f_60a75110d764.slice/crio-9bf142e2f20179006f748e6798d0c2f95d670472af66e9537351839e4c1aaa45 WatchSource:0}: Error finding container 9bf142e2f20179006f748e6798d0c2f95d670472af66e9537351839e4c1aaa45: Status 404 returned error can't find the container with id 9bf142e2f20179006f748e6798d0c2f95d670472af66e9537351839e4c1aaa45 Nov 01 22:22:41.671156 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:41.671131 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f0ded2e_e6aa_4764_a96b_ff2f41ce54b8.slice/crio-e23e2763a3591380a493e03e4209f502c432d0e813a2fcfb32604c549226ec42 WatchSource:0}: Error finding container e23e2763a3591380a493e03e4209f502c432d0e813a2fcfb32604c549226ec42: Status 404 returned error can't find the container with id e23e2763a3591380a493e03e4209f502c432d0e813a2fcfb32604c549226ec42 Nov 01 22:22:41.718009 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.717973 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:22:41.728307 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:41.728273 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d5cfa6_bf31_4fad_8003_d28fe9a37183.slice/crio-655cd6a658fa393611ebf9a14fa3a90a105d9e3e7f27469a5ee7addd752db01d WatchSource:0}: Error finding container 655cd6a658fa393611ebf9a14fa3a90a105d9e3e7f27469a5ee7addd752db01d: Status 404 returned error can't find the container with id 655cd6a658fa393611ebf9a14fa3a90a105d9e3e7f27469a5ee7addd752db01d Nov 01 22:22:41.859936 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:41.859891 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod179a164c_f6dd_42f9_9428_f241f81afff5.slice/crio-f0222f39630c858fec04d8f23a314c7c5687f9e0d827064de1b96d54d4bccdeb WatchSource:0}: Error finding container f0222f39630c858fec04d8f23a314c7c5687f9e0d827064de1b96d54d4bccdeb: Status 404 returned error can't find the container with id f0222f39630c858fec04d8f23a314c7c5687f9e0d827064de1b96d54d4bccdeb Nov 01 22:22:41.872341 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:22:41.871108 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bed4cac_d251_49be_9eb6_ce5cdb6852de.slice/crio-95acb05c8e561a0f8d3fe7df6dddd4812cb7042898b98ff64a446d0f4329b4c3 WatchSource:0}: Error finding container 95acb05c8e561a0f8d3fe7df6dddd4812cb7042898b98ff64a446d0f4329b4c3: Status 404 returned error can't find the container with id 95acb05c8e561a0f8d3fe7df6dddd4812cb7042898b98ff64a446d0f4329b4c3 Nov 01 22:22:41.876297 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.876273 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa603801-6c30-4cf1-ace8-30c8723e4a1e-original-pull-secret\") pod \"global-pull-secret-syncer-ttjsz\" (UID: \"fa603801-6c30-4cf1-ace8-30c8723e4a1e\") " pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:22:41.876452 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:41.876434 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:41.876502 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:41.876493 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa603801-6c30-4cf1-ace8-30c8723e4a1e-original-pull-secret podName:fa603801-6c30-4cf1-ace8-30c8723e4a1e nodeName:}" failed. No retries permitted until 2026-11-01 22:22:42.876478087 +0000 UTC m=+3.050287192 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa603801-6c30-4cf1-ace8-30c8723e4a1e-original-pull-secret") pod "global-pull-secret-syncer-ttjsz" (UID: "fa603801-6c30-4cf1-ace8-30c8723e4a1e") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:41.977337 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.977254 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc3de28d-1139-4677-81db-a114c8867933-metrics-certs\") pod \"network-metrics-daemon-xvgrl\" (UID: \"cc3de28d-1139-4677-81db-a114c8867933\") " pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:22:41.977337 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:41.977300 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxhl8\" (UniqueName: \"kubernetes.io/projected/b439b405-8bca-4ce8-980d-a076cb2f11bf-kube-api-access-lxhl8\") pod \"network-check-target-cfnmp\" (UID: \"b439b405-8bca-4ce8-980d-a076cb2f11bf\") " pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:22:41.977514 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:41.977421 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:41.977514 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:41.977431 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:22:41.977514 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:41.977448 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:22:41.977514 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:41.977458 2576 projected.go:196] Error preparing data for projected volume kube-api-access-lxhl8 for pod openshift-network-diagnostics/network-check-target-cfnmp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:41.977514 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:41.977491 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc3de28d-1139-4677-81db-a114c8867933-metrics-certs podName:cc3de28d-1139-4677-81db-a114c8867933 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:42.977476517 +0000 UTC m=+3.151285633 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc3de28d-1139-4677-81db-a114c8867933-metrics-certs") pod "network-metrics-daemon-xvgrl" (UID: "cc3de28d-1139-4677-81db-a114c8867933") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:41.977514 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:41.977508 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b439b405-8bca-4ce8-980d-a076cb2f11bf-kube-api-access-lxhl8 podName:b439b405-8bca-4ce8-980d-a076cb2f11bf nodeName:}" failed. No retries permitted until 2026-11-01 22:22:42.977500374 +0000 UTC m=+3.151309475 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxhl8" (UniqueName: "kubernetes.io/projected/b439b405-8bca-4ce8-980d-a076cb2f11bf-kube-api-access-lxhl8") pod "network-check-target-cfnmp" (UID: "b439b405-8bca-4ce8-980d-a076cb2f11bf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:42.193226 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:42.193191 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:22:42.300528 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:42.300410 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 22:17:41 +0000 UTC" deadline="2028-07-19 11:53:24.888005958 +0000 UTC" Nov 01 22:22:42.300528 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:42.300479 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15013h30m42.587531354s" Nov 01 22:22:42.403882 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:42.403843 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:22:42.404074 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:42.404030 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xvgrl" podUID="cc3de28d-1139-4677-81db-a114c8867933" Nov 01 22:22:42.416982 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:42.416904 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tmhm2" event={"ID":"3bed4cac-d251-49be-9eb6-ce5cdb6852de","Type":"ContainerStarted","Data":"95acb05c8e561a0f8d3fe7df6dddd4812cb7042898b98ff64a446d0f4329b4c3"} Nov 01 22:22:42.426104 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:42.426065 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-84vvx" event={"ID":"179a164c-f6dd-42f9-9428-f241f81afff5","Type":"ContainerStarted","Data":"f0222f39630c858fec04d8f23a314c7c5687f9e0d827064de1b96d54d4bccdeb"} Nov 01 22:22:42.435382 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:42.435345 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6h899" event={"ID":"45d5cfa6-bf31-4fad-8003-d28fe9a37183","Type":"ContainerStarted","Data":"655cd6a658fa393611ebf9a14fa3a90a105d9e3e7f27469a5ee7addd752db01d"} Nov 01 22:22:42.446178 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:42.446141 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-npsp2" event={"ID":"3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8","Type":"ContainerStarted","Data":"e23e2763a3591380a493e03e4209f502c432d0e813a2fcfb32604c549226ec42"} Nov 01 22:22:42.456277 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:42.456238 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" event={"ID":"14cdd491-7383-4cc5-ac4f-60a75110d764","Type":"ContainerStarted","Data":"9bf142e2f20179006f748e6798d0c2f95d670472af66e9537351839e4c1aaa45"} Nov 01 22:22:42.463970 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:42.463817 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jngxk" event={"ID":"3af15ebe-c67a-40c5-bac0-24410542413b","Type":"ContainerStarted","Data":"2f3d84313e3ca3144e9101369e8768171707b39c1a5069cface47f369f261a6e"} Nov 01 22:22:42.467367 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:42.467339 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rkf2j" event={"ID":"682b0c5d-fa6f-4a17-8b72-a1f224cf5fec","Type":"ContainerStarted","Data":"0dc901321c676181e22a19ae8becbeb76d04ef9b84ee2856e6a59c9ffffbbae2"} Nov 01 22:22:42.503544 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:42.503505 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" event={"ID":"5c959caa-7239-4b81-8c19-c867fc6c48e6","Type":"ContainerStarted","Data":"a2d0ec52d05829919a9fe2e609c56582be559d0438c6ddc983191adb7751a8ce"} Nov 01 22:22:42.518635 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:42.518593 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x4k8b" event={"ID":"f71460db-658f-45ed-980f-da65509e2bc5","Type":"ContainerStarted","Data":"abacbe9c358eb7c91c09dbba58a8c965c778bac8677000afff60f2a0ecc3275b"} Nov 01 22:22:42.533320 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:42.533167 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal" event={"ID":"fbc5f96efa48930e21a66fef652c693e","Type":"ContainerStarted","Data":"1d8b4c5a4a30cc4854e7d670f40126715432e1fba59c5f74cf836857393e1784"} Nov 01 22:22:42.557723 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:42.557179 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-238.us-east-2.compute.internal" event={"ID":"4f1e77a94f837e0f1b12b6a1e77eab89","Type":"ContainerStarted","Data":"caa1cde0c045e60f020bda4fa2b9c477955b5b0f1ed5dda0dc33c7b8df8916b6"} Nov 01 22:22:42.647699 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:42.647632 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:22:42.887220 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:42.887087 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa603801-6c30-4cf1-ace8-30c8723e4a1e-original-pull-secret\") pod \"global-pull-secret-syncer-ttjsz\" (UID: \"fa603801-6c30-4cf1-ace8-30c8723e4a1e\") " pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:22:42.887392 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:42.887251 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:42.887392 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:42.887329 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa603801-6c30-4cf1-ace8-30c8723e4a1e-original-pull-secret podName:fa603801-6c30-4cf1-ace8-30c8723e4a1e nodeName:}" failed. No retries permitted until 2026-11-01 22:22:44.887305733 +0000 UTC m=+5.061114839 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa603801-6c30-4cf1-ace8-30c8723e4a1e-original-pull-secret") pod "global-pull-secret-syncer-ttjsz" (UID: "fa603801-6c30-4cf1-ace8-30c8723e4a1e") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:42.987698 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:42.987655 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc3de28d-1139-4677-81db-a114c8867933-metrics-certs\") pod \"network-metrics-daemon-xvgrl\" (UID: \"cc3de28d-1139-4677-81db-a114c8867933\") " pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:22:42.987954 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:42.987715 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxhl8\" (UniqueName: \"kubernetes.io/projected/b439b405-8bca-4ce8-980d-a076cb2f11bf-kube-api-access-lxhl8\") pod \"network-check-target-cfnmp\" (UID: \"b439b405-8bca-4ce8-980d-a076cb2f11bf\") " pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:22:42.988051 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:42.988002 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:22:42.988051 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:42.988029 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:22:42.988051 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:42.988046 2576 projected.go:196] Error preparing data for projected volume kube-api-access-lxhl8 for pod openshift-network-diagnostics/network-check-target-cfnmp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:42.988236 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:42.988115 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b439b405-8bca-4ce8-980d-a076cb2f11bf-kube-api-access-lxhl8 podName:b439b405-8bca-4ce8-980d-a076cb2f11bf nodeName:}" failed. No retries permitted until 2026-11-01 22:22:44.988093402 +0000 UTC m=+5.161902512 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxhl8" (UniqueName: "kubernetes.io/projected/b439b405-8bca-4ce8-980d-a076cb2f11bf-kube-api-access-lxhl8") pod "network-check-target-cfnmp" (UID: "b439b405-8bca-4ce8-980d-a076cb2f11bf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:42.988775 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:42.988749 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:42.988898 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:42.988815 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc3de28d-1139-4677-81db-a114c8867933-metrics-certs podName:cc3de28d-1139-4677-81db-a114c8867933 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:44.988798317 +0000 UTC m=+5.162607446 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc3de28d-1139-4677-81db-a114c8867933-metrics-certs") pod "network-metrics-daemon-xvgrl" (UID: "cc3de28d-1139-4677-81db-a114c8867933") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:43.300870 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:43.300808 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 22:17:41 +0000 UTC" deadline="2028-05-23 17:08:33.718002125 +0000 UTC" Nov 01 22:22:43.300870 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:43.300867 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13650h45m50.417140063s" Nov 01 22:22:43.401608 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:43.401573 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:22:43.401863 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:43.401710 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnmp" podUID="b439b405-8bca-4ce8-980d-a076cb2f11bf" Nov 01 22:22:43.402339 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:43.402314 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:22:43.402464 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:43.402439 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttjsz" podUID="fa603801-6c30-4cf1-ace8-30c8723e4a1e" Nov 01 22:22:44.405574 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:44.405130 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:22:44.405574 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:44.405297 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xvgrl" podUID="cc3de28d-1139-4677-81db-a114c8867933" Nov 01 22:22:44.905330 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:44.904778 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa603801-6c30-4cf1-ace8-30c8723e4a1e-original-pull-secret\") pod \"global-pull-secret-syncer-ttjsz\" (UID: \"fa603801-6c30-4cf1-ace8-30c8723e4a1e\") " pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:22:44.905330 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:44.904944 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:44.905330 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:44.905010 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa603801-6c30-4cf1-ace8-30c8723e4a1e-original-pull-secret podName:fa603801-6c30-4cf1-ace8-30c8723e4a1e nodeName:}" failed. No retries permitted until 2026-11-01 22:22:48.904991738 +0000 UTC m=+9.078800845 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa603801-6c30-4cf1-ace8-30c8723e4a1e-original-pull-secret") pod "global-pull-secret-syncer-ttjsz" (UID: "fa603801-6c30-4cf1-ace8-30c8723e4a1e") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:45.006440 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:45.005441 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc3de28d-1139-4677-81db-a114c8867933-metrics-certs\") pod \"network-metrics-daemon-xvgrl\" (UID: \"cc3de28d-1139-4677-81db-a114c8867933\") " pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:22:45.006440 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:45.005509 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxhl8\" (UniqueName: \"kubernetes.io/projected/b439b405-8bca-4ce8-980d-a076cb2f11bf-kube-api-access-lxhl8\") pod \"network-check-target-cfnmp\" (UID: \"b439b405-8bca-4ce8-980d-a076cb2f11bf\") " pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:22:45.006440 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:45.005710 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:22:45.006440 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:45.005733 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:22:45.006440 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:45.005748 2576 projected.go:196] Error preparing data for projected volume kube-api-access-lxhl8 for pod openshift-network-diagnostics/network-check-target-cfnmp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:45.006440 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:45.005809 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b439b405-8bca-4ce8-980d-a076cb2f11bf-kube-api-access-lxhl8 podName:b439b405-8bca-4ce8-980d-a076cb2f11bf nodeName:}" failed. No retries permitted until 2026-11-01 22:22:49.005789236 +0000 UTC m=+9.179598344 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxhl8" (UniqueName: "kubernetes.io/projected/b439b405-8bca-4ce8-980d-a076cb2f11bf-kube-api-access-lxhl8") pod "network-check-target-cfnmp" (UID: "b439b405-8bca-4ce8-980d-a076cb2f11bf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:45.006440 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:45.005996 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:45.006440 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:45.006042 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc3de28d-1139-4677-81db-a114c8867933-metrics-certs podName:cc3de28d-1139-4677-81db-a114c8867933 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:49.006026833 +0000 UTC m=+9.179835941 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc3de28d-1139-4677-81db-a114c8867933-metrics-certs") pod "network-metrics-daemon-xvgrl" (UID: "cc3de28d-1139-4677-81db-a114c8867933") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:45.401684 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:45.401023 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:22:45.401684 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:45.401171 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnmp" podUID="b439b405-8bca-4ce8-980d-a076cb2f11bf" Nov 01 22:22:45.401684 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:45.401023 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:22:45.401684 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:45.401283 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttjsz" podUID="fa603801-6c30-4cf1-ace8-30c8723e4a1e" Nov 01 22:22:46.404632 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:46.404315 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:22:46.405155 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:46.404717 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xvgrl" podUID="cc3de28d-1139-4677-81db-a114c8867933" Nov 01 22:22:46.613422 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:46.613384 2576 generic.go:356] "Generic (PLEG): container finished" podID="3af15ebe-c67a-40c5-bac0-24410542413b" containerID="687214760f8a2c4c662547487e75245b6a52aa09487e8a84f4644c7257c5e5ef" exitCode=0 Nov 01 22:22:46.614041 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:46.613992 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jngxk" event={"ID":"3af15ebe-c67a-40c5-bac0-24410542413b","Type":"ContainerDied","Data":"687214760f8a2c4c662547487e75245b6a52aa09487e8a84f4644c7257c5e5ef"} Nov 01 22:22:46.621575 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:46.621450 2576 generic.go:356] "Generic (PLEG): container finished" podID="fbc5f96efa48930e21a66fef652c693e" containerID="6d07763750c179a74435723f3dd52c79ce6ba457ef0de02540c5b793d0a7fcca" exitCode=0 Nov 01 22:22:46.621575 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:46.621498 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal" event={"ID":"fbc5f96efa48930e21a66fef652c693e","Type":"ContainerDied","Data":"6d07763750c179a74435723f3dd52c79ce6ba457ef0de02540c5b793d0a7fcca"} Nov 01 22:22:47.401360 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:47.401273 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:22:47.401526 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:47.401422 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttjsz" podUID="fa603801-6c30-4cf1-ace8-30c8723e4a1e" Nov 01 22:22:47.401940 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:47.401919 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:22:47.402050 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:47.402029 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnmp" podUID="b439b405-8bca-4ce8-980d-a076cb2f11bf" Nov 01 22:22:47.629621 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:47.629578 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tmhm2" event={"ID":"3bed4cac-d251-49be-9eb6-ce5cdb6852de","Type":"ContainerStarted","Data":"b99819c72a2ec2912346313d02fdd7447d050e30f88fc2e1235f31f870cdab54"} Nov 01 22:22:47.635234 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:47.635203 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-238.us-east-2.compute.internal" event={"ID":"4f1e77a94f837e0f1b12b6a1e77eab89","Type":"ContainerStarted","Data":"8d65edf964dbf913fbee970830bb966eb40d9348b49138573cc5fd5f1a457395"} Nov 01 22:22:47.662921 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:47.662809 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tmhm2" podStartSLOduration=1.514211624 podStartE2EDuration="6.662786647s" totalImagesPullingTime="5.148575023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:47.646701056 +0000 UTC m=+7.820510182" watchObservedRunningTime="2026-11-01 22:22:47.662786647 +0000 UTC m=+7.836595774" Nov 01 22:22:48.401451 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:48.401413 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:22:48.401637 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:48.401567 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xvgrl" podUID="cc3de28d-1139-4677-81db-a114c8867933" Nov 01 22:22:48.644639 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:48.643958 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rkf2j" event={"ID":"682b0c5d-fa6f-4a17-8b72-a1f224cf5fec","Type":"ContainerStarted","Data":"acee695a428ebbbc940999ed86eb2ed1572440f54133f48a304fa4208a9715b4"} Nov 01 22:22:48.649635 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:48.649577 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal" event={"ID":"fbc5f96efa48930e21a66fef652c693e","Type":"ContainerStarted","Data":"687172ba2729d825e69e7fea1c2fa891ae7c7810ddec386070b79393dcf4182d"} Nov 01 22:22:48.660187 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:48.659444 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-238.us-east-2.compute.internal" podStartSLOduration=7.6594255780000005 podStartE2EDuration="7.659425578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:47.678485914 +0000 UTC m=+7.852295043" watchObservedRunningTime="2026-11-01 22:22:48.659425578 +0000 UTC m=+8.833234707" Nov 01 22:22:48.678074 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:48.677158 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal" podStartSLOduration=7.677140164 podStartE2EDuration="7.677140164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:48.676960262 +0000 UTC m=+8.850769385" watchObservedRunningTime="2026-11-01 22:22:48.677140164 +0000 UTC m=+8.850949288" Nov 01 22:22:48.678074 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:48.677319 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rkf2j" podStartSLOduration=2.733617858 podStartE2EDuration="8.677309135s" totalImagesPullingTime="5.943691277s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:48.659952819 +0000 UTC m=+8.833761943" watchObservedRunningTime="2026-11-01 22:22:48.677309135 +0000 UTC m=+8.851118258" Nov 01 22:22:48.937610 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:48.937570 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa603801-6c30-4cf1-ace8-30c8723e4a1e-original-pull-secret\") pod \"global-pull-secret-syncer-ttjsz\" (UID: \"fa603801-6c30-4cf1-ace8-30c8723e4a1e\") " pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:22:48.937801 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:48.937710 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:48.937801 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:48.937775 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa603801-6c30-4cf1-ace8-30c8723e4a1e-original-pull-secret podName:fa603801-6c30-4cf1-ace8-30c8723e4a1e nodeName:}" failed. No retries permitted until 2026-11-01 22:22:56.937753685 +0000 UTC m=+17.111562787 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa603801-6c30-4cf1-ace8-30c8723e4a1e-original-pull-secret") pod "global-pull-secret-syncer-ttjsz" (UID: "fa603801-6c30-4cf1-ace8-30c8723e4a1e") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:49.038366 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:49.038323 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc3de28d-1139-4677-81db-a114c8867933-metrics-certs\") pod \"network-metrics-daemon-xvgrl\" (UID: \"cc3de28d-1139-4677-81db-a114c8867933\") " pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:22:49.038563 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:49.038380 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxhl8\" (UniqueName: \"kubernetes.io/projected/b439b405-8bca-4ce8-980d-a076cb2f11bf-kube-api-access-lxhl8\") pod \"network-check-target-cfnmp\" (UID: \"b439b405-8bca-4ce8-980d-a076cb2f11bf\") " pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:22:49.038563 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:49.038483 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:49.038673 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:49.038562 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:22:49.038673 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:49.038581 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:22:49.038673 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:49.038594 2576 projected.go:196] Error preparing data for projected volume kube-api-access-lxhl8 for pod openshift-network-diagnostics/network-check-target-cfnmp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:49.038673 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:49.038565 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc3de28d-1139-4677-81db-a114c8867933-metrics-certs podName:cc3de28d-1139-4677-81db-a114c8867933 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:57.038538549 +0000 UTC m=+17.212347664 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc3de28d-1139-4677-81db-a114c8867933-metrics-certs") pod "network-metrics-daemon-xvgrl" (UID: "cc3de28d-1139-4677-81db-a114c8867933") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:49.038673 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:49.038664 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b439b405-8bca-4ce8-980d-a076cb2f11bf-kube-api-access-lxhl8 podName:b439b405-8bca-4ce8-980d-a076cb2f11bf nodeName:}" failed. No retries permitted until 2026-11-01 22:22:57.038647745 +0000 UTC m=+17.212456850 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxhl8" (UniqueName: "kubernetes.io/projected/b439b405-8bca-4ce8-980d-a076cb2f11bf-kube-api-access-lxhl8") pod "network-check-target-cfnmp" (UID: "b439b405-8bca-4ce8-980d-a076cb2f11bf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:49.359863 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:49.359737 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tmhm2" Nov 01 22:22:49.360371 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:49.360347 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tmhm2" Nov 01 22:22:49.400876 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:49.400822 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:22:49.401062 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:49.400982 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnmp" podUID="b439b405-8bca-4ce8-980d-a076cb2f11bf" Nov 01 22:22:49.401392 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:49.401372 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:22:49.401476 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:49.401462 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttjsz" podUID="fa603801-6c30-4cf1-ace8-30c8723e4a1e" Nov 01 22:22:49.657844 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:49.657754 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" event={"ID":"14cdd491-7383-4cc5-ac4f-60a75110d764","Type":"ContainerStarted","Data":"9029eda7b62f9a899cacc0ee4159177fecb37dd99136a95a6807241373547fe0"} Nov 01 22:22:50.402910 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:50.402885 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:22:50.403059 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:50.403004 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xvgrl" podUID="cc3de28d-1139-4677-81db-a114c8867933" Nov 01 22:22:50.475733 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:50.474880 2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 22:22:50.662068 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:50.662029 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-84vvx" event={"ID":"179a164c-f6dd-42f9-9428-f241f81afff5","Type":"ContainerStarted","Data":"422241d6fd558ed5f071dfabbb1ab94c93eb281b78594e2403f6aa11b00c705a"} Nov 01 22:22:50.665313 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:50.665285 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" event={"ID":"14cdd491-7383-4cc5-ac4f-60a75110d764","Type":"ContainerStarted","Data":"05ad13156d4d8f7fbcfe3446e679edeacebb1407cde8734bde8761dfcdf1c3d2"} Nov 01 22:22:50.665498 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:50.665351 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 22:22:50.680136 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:50.680075 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-84vvx" podStartSLOduration=1.1153522 podStartE2EDuration="9.680057931s" totalImagesPullingTime="8.564705731s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:50.679954414 +0000 UTC m=+10.853763550" watchObservedRunningTime="2026-11-01 22:22:50.680057931 +0000 UTC m=+10.853867055" Nov 01 22:22:51.322939 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:51.322796 2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T22:22:50.474900296Z","UUID":"5a16245f-71bc-4edd-99f5-1d8468007015","Handler":null,"Name":"","Endpoint":""} Nov 01 22:22:51.324557 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:51.324538 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 22:22:51.324557 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:51.324563 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 22:22:51.401380 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:51.401346 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:22:51.401555 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:51.401527 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttjsz" podUID="fa603801-6c30-4cf1-ace8-30c8723e4a1e" Nov 01 22:22:51.401608 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:51.401570 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:22:51.401697 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:51.401675 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnmp" podUID="b439b405-8bca-4ce8-980d-a076cb2f11bf" Nov 01 22:22:51.670427 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:51.670389 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x4k8b" event={"ID":"f71460db-658f-45ed-980f-da65509e2bc5","Type":"ContainerStarted","Data":"0215d857e8597aecc1d9ccbb4795c5653f58fd62a9e93a374575254c2c6c11c5"} Nov 01 22:22:51.673045 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:51.673016 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" event={"ID":"14cdd491-7383-4cc5-ac4f-60a75110d764","Type":"ContainerStarted","Data":"22f9916e99db23f1361580fefb67cc0eca7a005f8ae61b5123b58839cabefd1b"} Nov 01 22:22:51.676075 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:51.675562 2576 generic.go:356] "Generic (PLEG): container finished" podID="3af15ebe-c67a-40c5-bac0-24410542413b" containerID="de4a86630cd89dae0d71ecbd02a0186f63d66dd7865628934190c5fa18d80294" exitCode=0 Nov 01 22:22:51.676075 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:51.675687 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jngxk" event={"ID":"3af15ebe-c67a-40c5-bac0-24410542413b","Type":"ContainerDied","Data":"de4a86630cd89dae0d71ecbd02a0186f63d66dd7865628934190c5fa18d80294"} Nov 01 22:22:51.685728 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:51.685666 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-x4k8b" podStartSLOduration=2.531470845 podStartE2EDuration="11.685650288s" totalImagesPullingTime="9.154179443s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:51.68540493 +0000 UTC m=+11.859214052" watchObservedRunningTime="2026-11-01 22:22:51.685650288 +0000 UTC m=+11.859459411" Nov 01 22:22:51.735139 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:51.735085 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" podStartSLOduration=2.337234891 podStartE2EDuration="11.735067054s" totalImagesPullingTime="9.397832163s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:51.734774423 +0000 UTC m=+11.908583549" watchObservedRunningTime="2026-11-01 22:22:51.735067054 +0000 UTC m=+11.908876175" Nov 01 22:22:52.404497 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:52.404266 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:22:52.404642 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:52.404615 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xvgrl" podUID="cc3de28d-1139-4677-81db-a114c8867933" Nov 01 22:22:52.678759 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:52.678725 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" event={"ID":"5c959caa-7239-4b81-8c19-c867fc6c48e6","Type":"ContainerStarted","Data":"74d2d7bb55b61fae911c4881d8269a3fed5efb0e50fa08a789cb64056908a608"} Nov 01 22:22:52.680440 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:52.680410 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-npsp2" event={"ID":"3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8","Type":"ContainerStarted","Data":"cafa0618b562cef0dfe8bb15e6edd89ee83827a95483b110a983b3cd0e48158a"} Nov 01 22:22:52.682901 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:52.682878 2576 generic.go:356] "Generic (PLEG): container finished" podID="3af15ebe-c67a-40c5-bac0-24410542413b" containerID="119ac3863d9b933467c639e15af6bc4b571e6a2be1ff835169d0a2047ee1945f" exitCode=0 Nov 01 22:22:52.683024 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:52.682929 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jngxk" event={"ID":"3af15ebe-c67a-40c5-bac0-24410542413b","Type":"ContainerDied","Data":"119ac3863d9b933467c639e15af6bc4b571e6a2be1ff835169d0a2047ee1945f"} Nov 01 22:22:52.695097 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:52.695000 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" podStartSLOduration=1.979703362 podStartE2EDuration="12.694985304s" totalImagesPullingTime="10.715281942s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:52.694915283 +0000 UTC m=+12.868724404" watchObservedRunningTime="2026-11-01 22:22:52.694985304 +0000 UTC m=+12.868794427" Nov 01 22:22:52.709261 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:52.709220 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-npsp2" podStartSLOduration=2.758938268 podStartE2EDuration="11.70920939s" totalImagesPullingTime="8.950271122s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:52.708865494 +0000 UTC m=+12.882674613" watchObservedRunningTime="2026-11-01 22:22:52.70920939 +0000 UTC m=+12.883018511" Nov 01 22:22:52.840044 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:52.840009 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tmhm2" Nov 01 22:22:52.840219 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:52.840157 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 22:22:52.840669 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:52.840649 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tmhm2" Nov 01 22:22:53.401557 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:53.401526 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:22:53.401761 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:53.401635 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnmp" podUID="b439b405-8bca-4ce8-980d-a076cb2f11bf" Nov 01 22:22:53.401761 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:53.401692 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:22:53.401879 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:53.401802 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttjsz" podUID="fa603801-6c30-4cf1-ace8-30c8723e4a1e" Nov 01 22:22:54.401210 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:54.401181 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:22:54.401741 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:54.401298 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xvgrl" podUID="cc3de28d-1139-4677-81db-a114c8867933" Nov 01 22:22:54.692658 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:54.692620 2576 generic.go:356] "Generic (PLEG): container finished" podID="3af15ebe-c67a-40c5-bac0-24410542413b" containerID="a61d238af5ad4bed3be78c4ad6a75ecdd29a2597f5eec7f03e8e1841704ab771" exitCode=0 Nov 01 22:22:54.692806 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:54.692668 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jngxk" event={"ID":"3af15ebe-c67a-40c5-bac0-24410542413b","Type":"ContainerDied","Data":"a61d238af5ad4bed3be78c4ad6a75ecdd29a2597f5eec7f03e8e1841704ab771"} Nov 01 22:22:55.401608 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:55.401574 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:22:55.402301 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:55.401701 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnmp" podUID="b439b405-8bca-4ce8-980d-a076cb2f11bf" Nov 01 22:22:55.402301 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:55.401758 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:22:55.402301 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:55.401866 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttjsz" podUID="fa603801-6c30-4cf1-ace8-30c8723e4a1e" Nov 01 22:22:56.401104 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:56.401059 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:22:56.401357 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:56.401334 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xvgrl" podUID="cc3de28d-1139-4677-81db-a114c8867933" Nov 01 22:22:57.003321 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:57.003279 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa603801-6c30-4cf1-ace8-30c8723e4a1e-original-pull-secret\") pod \"global-pull-secret-syncer-ttjsz\" (UID: \"fa603801-6c30-4cf1-ace8-30c8723e4a1e\") " pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:22:57.003879 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:57.003444 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:57.003879 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:57.003534 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa603801-6c30-4cf1-ace8-30c8723e4a1e-original-pull-secret podName:fa603801-6c30-4cf1-ace8-30c8723e4a1e nodeName:}" failed. No retries permitted until 2026-11-01 22:23:13.003508277 +0000 UTC m=+33.177317380 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa603801-6c30-4cf1-ace8-30c8723e4a1e-original-pull-secret") pod "global-pull-secret-syncer-ttjsz" (UID: "fa603801-6c30-4cf1-ace8-30c8723e4a1e") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:57.104987 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:57.104669 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc3de28d-1139-4677-81db-a114c8867933-metrics-certs\") pod \"network-metrics-daemon-xvgrl\" (UID: \"cc3de28d-1139-4677-81db-a114c8867933\") " pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:22:57.104987 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:57.104718 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxhl8\" (UniqueName: \"kubernetes.io/projected/b439b405-8bca-4ce8-980d-a076cb2f11bf-kube-api-access-lxhl8\") pod \"network-check-target-cfnmp\" (UID: \"b439b405-8bca-4ce8-980d-a076cb2f11bf\") " pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:22:57.104987 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:57.104825 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:57.104987 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:57.104899 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:22:57.104987 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:57.104912 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc3de28d-1139-4677-81db-a114c8867933-metrics-certs podName:cc3de28d-1139-4677-81db-a114c8867933 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:13.104896051 +0000 UTC m=+33.278705170 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc3de28d-1139-4677-81db-a114c8867933-metrics-certs") pod "network-metrics-daemon-xvgrl" (UID: "cc3de28d-1139-4677-81db-a114c8867933") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:57.104987 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:57.104913 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:22:57.104987 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:57.104926 2576 projected.go:196] Error preparing data for projected volume kube-api-access-lxhl8 for pod openshift-network-diagnostics/network-check-target-cfnmp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:57.104987 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:57.104961 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b439b405-8bca-4ce8-980d-a076cb2f11bf-kube-api-access-lxhl8 podName:b439b405-8bca-4ce8-980d-a076cb2f11bf nodeName:}" failed. No retries permitted until 2026-11-01 22:23:13.104950518 +0000 UTC m=+33.278759637 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxhl8" (UniqueName: "kubernetes.io/projected/b439b405-8bca-4ce8-980d-a076cb2f11bf-kube-api-access-lxhl8") pod "network-check-target-cfnmp" (UID: "b439b405-8bca-4ce8-980d-a076cb2f11bf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:57.401032 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:57.401002 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:22:57.401032 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:57.401036 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:22:57.401187 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:57.401116 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnmp" podUID="b439b405-8bca-4ce8-980d-a076cb2f11bf" Nov 01 22:22:57.401241 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:57.401176 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttjsz" podUID="fa603801-6c30-4cf1-ace8-30c8723e4a1e" Nov 01 22:22:57.702237 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:57.702204 2576 generic.go:356] "Generic (PLEG): container finished" podID="3af15ebe-c67a-40c5-bac0-24410542413b" containerID="e7dddd97aa426a465a3958cc8bf2636f5c91fc2b102bd9d625d5ef5aa0c3340b" exitCode=0 Nov 01 22:22:57.702415 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:57.702265 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jngxk" event={"ID":"3af15ebe-c67a-40c5-bac0-24410542413b","Type":"ContainerDied","Data":"e7dddd97aa426a465a3958cc8bf2636f5c91fc2b102bd9d625d5ef5aa0c3340b"} Nov 01 22:22:58.401765 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:58.401541 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:22:58.402244 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:58.401887 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xvgrl" podUID="cc3de28d-1139-4677-81db-a114c8867933" Nov 01 22:22:58.707816 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:58.707780 2576 generic.go:356] "Generic (PLEG): container finished" podID="3af15ebe-c67a-40c5-bac0-24410542413b" containerID="75629e491d6b06c975de9654b36e8e5f27055e05c4ce39087e8e98bec9b88eb5" exitCode=0 Nov 01 22:22:58.708038 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:58.707859 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jngxk" event={"ID":"3af15ebe-c67a-40c5-bac0-24410542413b","Type":"ContainerDied","Data":"75629e491d6b06c975de9654b36e8e5f27055e05c4ce39087e8e98bec9b88eb5"} Nov 01 22:22:59.401388 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:59.401352 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:22:59.401388 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:59.401399 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:22:59.401628 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:59.401492 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnmp" podUID="b439b405-8bca-4ce8-980d-a076cb2f11bf" Nov 01 22:22:59.401681 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:22:59.401606 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttjsz" podUID="fa603801-6c30-4cf1-ace8-30c8723e4a1e" Nov 01 22:22:59.715026 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:59.714991 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jngxk" event={"ID":"3af15ebe-c67a-40c5-bac0-24410542413b","Type":"ContainerStarted","Data":"03d225509b7867cc7f6f2166a1ee274587a3ea7541aa348f8ee58d5efce93d0e"} Nov 01 22:22:59.738645 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:22:59.738594 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jngxk" podStartSLOduration=8.153142756 podStartE2EDuration="19.738576036s" totalImagesPullingTime="11.58543328s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:59.737893146 +0000 UTC m=+19.911702268" watchObservedRunningTime="2026-11-01 22:22:59.738576036 +0000 UTC m=+19.912385162" Nov 01 22:23:00.404503 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:00.404472 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:23:00.404721 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:00.404613 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xvgrl" podUID="cc3de28d-1139-4677-81db-a114c8867933" Nov 01 22:23:01.401316 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:01.401289 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:23:01.401642 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:01.401318 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:23:01.401642 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:01.401426 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnmp" podUID="b439b405-8bca-4ce8-980d-a076cb2f11bf" Nov 01 22:23:01.401642 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:01.401521 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttjsz" podUID="fa603801-6c30-4cf1-ace8-30c8723e4a1e" Nov 01 22:23:01.722620 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:01.722370 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6h899" event={"ID":"45d5cfa6-bf31-4fad-8003-d28fe9a37183","Type":"ContainerStarted","Data":"317482d952a31c2744864c8c003b6bcab2941c1db54f0e2a3b0787874fc69ac5"} Nov 01 22:23:01.722777 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:01.722620 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6h899" event={"ID":"45d5cfa6-bf31-4fad-8003-d28fe9a37183","Type":"ContainerStarted","Data":"49b71b0bdc401bfb2534ab33d47dc0065efa810b614bfc291651361b4017ede8"} Nov 01 22:23:01.722777 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:01.722665 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6h899" event={"ID":"45d5cfa6-bf31-4fad-8003-d28fe9a37183","Type":"ContainerStarted","Data":"a7e75c8bcbca577f8f492797f576a6e7d45401c0a92f84b533329eb7139ab71d"} Nov 01 22:23:01.722777 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:01.722703 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6h899" event={"ID":"45d5cfa6-bf31-4fad-8003-d28fe9a37183","Type":"ContainerStarted","Data":"6d38b24a00d22eb41c9f4f0c493168f4aa56f3592e34b7deb1062c40f12ec0a6"} Nov 01 22:23:01.722777 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:01.722743 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6h899" event={"ID":"45d5cfa6-bf31-4fad-8003-d28fe9a37183","Type":"ContainerStarted","Data":"4777bbc1b3767c9e66095ec002c0b8d598cedf2937d9ae21b4b5ee977d4ff3c1"} Nov 01 22:23:01.722953 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:01.722784 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6h899" event={"ID":"45d5cfa6-bf31-4fad-8003-d28fe9a37183","Type":"ContainerStarted","Data":"562ae17d5065c5e6e73a8b06e263b03cf24a3c83bef42f0c4faaf539c48be1fb"} Nov 01 22:23:02.401697 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:02.401668 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:23:02.402285 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:02.401810 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xvgrl" podUID="cc3de28d-1139-4677-81db-a114c8867933" Nov 01 22:23:03.401202 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:03.401166 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:23:03.401202 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:03.401190 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:23:03.401470 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:03.401286 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnmp" podUID="b439b405-8bca-4ce8-980d-a076cb2f11bf" Nov 01 22:23:03.401470 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:03.401403 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttjsz" podUID="fa603801-6c30-4cf1-ace8-30c8723e4a1e" Nov 01 22:23:03.730272 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:03.730235 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6h899" event={"ID":"45d5cfa6-bf31-4fad-8003-d28fe9a37183","Type":"ContainerStarted","Data":"b125bd8b33b6761c93acf11562e7bf9ea101a1ebc245810450a465cc8f4bc1a2"} Nov 01 22:23:04.403742 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:04.403716 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:23:04.403951 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:04.403862 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xvgrl" podUID="cc3de28d-1139-4677-81db-a114c8867933" Nov 01 22:23:05.401699 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:05.401659 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:23:05.401699 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:05.401686 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:23:05.402245 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:05.401778 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttjsz" podUID="fa603801-6c30-4cf1-ace8-30c8723e4a1e" Nov 01 22:23:05.402245 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:05.401864 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnmp" podUID="b439b405-8bca-4ce8-980d-a076cb2f11bf" Nov 01 22:23:05.738693 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:05.738659 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6h899" event={"ID":"45d5cfa6-bf31-4fad-8003-d28fe9a37183","Type":"ContainerStarted","Data":"1c3fd98c82b0d0fdaceceb4dd665d07e61ace03f8341cb2dd55e346d9e404a90"} Nov 01 22:23:05.739064 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:05.739040 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:05.739209 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:05.739116 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:05.739209 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:05.739171 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:05.755196 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:05.755172 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:05.755822 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:05.755806 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:05.770909 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:05.770862 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6h899" podStartSLOduration=5.327348195 podStartE2EDuration="24.770814319s" totalImagesPullingTime="19.443466124s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:05.770305934 +0000 UTC m=+25.944115056" watchObservedRunningTime="2026-11-01 22:23:05.770814319 +0000 UTC m=+25.944623440" Nov 01 22:23:06.400951 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:06.400913 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:23:06.401102 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:06.401057 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xvgrl" podUID="cc3de28d-1139-4677-81db-a114c8867933" Nov 01 22:23:07.314390 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:07.314350 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cfnmp"] Nov 01 22:23:07.315352 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:07.314513 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:23:07.315352 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:07.314602 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnmp" podUID="b439b405-8bca-4ce8-980d-a076cb2f11bf" Nov 01 22:23:07.317717 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:07.317692 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ttjsz"] Nov 01 22:23:07.318070 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:07.318047 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:23:07.318193 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:07.318173 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttjsz" podUID="fa603801-6c30-4cf1-ace8-30c8723e4a1e" Nov 01 22:23:07.318441 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:07.318421 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xvgrl"] Nov 01 22:23:07.318556 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:07.318538 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:23:07.318667 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:07.318641 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xvgrl" podUID="cc3de28d-1139-4677-81db-a114c8867933" Nov 01 22:23:09.401589 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:09.401561 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:23:09.401975 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:09.401652 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnmp" podUID="b439b405-8bca-4ce8-980d-a076cb2f11bf" Nov 01 22:23:09.401975 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:09.401727 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:23:09.401975 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:09.401814 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttjsz" podUID="fa603801-6c30-4cf1-ace8-30c8723e4a1e" Nov 01 22:23:09.401975 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:09.401871 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:23:09.401975 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:09.401925 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xvgrl" podUID="cc3de28d-1139-4677-81db-a114c8867933" Nov 01 22:23:11.401900 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:11.401686 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:23:11.402272 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:11.401705 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:23:11.402272 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:11.401981 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnmp" podUID="b439b405-8bca-4ce8-980d-a076cb2f11bf" Nov 01 22:23:11.402272 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:11.402066 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xvgrl" podUID="cc3de28d-1139-4677-81db-a114c8867933" Nov 01 22:23:11.402272 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:11.401761 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:23:11.402272 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:11.402153 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttjsz" podUID="fa603801-6c30-4cf1-ace8-30c8723e4a1e" Nov 01 22:23:13.017271 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.017231 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa603801-6c30-4cf1-ace8-30c8723e4a1e-original-pull-secret\") pod \"global-pull-secret-syncer-ttjsz\" (UID: \"fa603801-6c30-4cf1-ace8-30c8723e4a1e\") " pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:23:13.017764 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:13.017372 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:23:13.017764 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:13.017446 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa603801-6c30-4cf1-ace8-30c8723e4a1e-original-pull-secret podName:fa603801-6c30-4cf1-ace8-30c8723e4a1e nodeName:}" failed. No retries permitted until 2026-11-01 22:23:45.017426492 +0000 UTC m=+65.191235612 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa603801-6c30-4cf1-ace8-30c8723e4a1e-original-pull-secret") pod "global-pull-secret-syncer-ttjsz" (UID: "fa603801-6c30-4cf1-ace8-30c8723e4a1e") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:23:13.117768 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.117676 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc3de28d-1139-4677-81db-a114c8867933-metrics-certs\") pod \"network-metrics-daemon-xvgrl\" (UID: \"cc3de28d-1139-4677-81db-a114c8867933\") " pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:23:13.117768 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.117724 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxhl8\" (UniqueName: \"kubernetes.io/projected/b439b405-8bca-4ce8-980d-a076cb2f11bf-kube-api-access-lxhl8\") pod \"network-check-target-cfnmp\" (UID: \"b439b405-8bca-4ce8-980d-a076cb2f11bf\") " pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:23:13.118018 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:13.117819 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:23:13.118018 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:13.117874 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:23:13.118018 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:13.117894 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:23:13.118018 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:13.117901 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc3de28d-1139-4677-81db-a114c8867933-metrics-certs podName:cc3de28d-1139-4677-81db-a114c8867933 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:45.117885504 +0000 UTC m=+65.291694605 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc3de28d-1139-4677-81db-a114c8867933-metrics-certs") pod "network-metrics-daemon-xvgrl" (UID: "cc3de28d-1139-4677-81db-a114c8867933") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:23:13.118018 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:13.117907 2576 projected.go:196] Error preparing data for projected volume kube-api-access-lxhl8 for pod openshift-network-diagnostics/network-check-target-cfnmp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:23:13.118018 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:13.117954 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b439b405-8bca-4ce8-980d-a076cb2f11bf-kube-api-access-lxhl8 podName:b439b405-8bca-4ce8-980d-a076cb2f11bf nodeName:}" failed. No retries permitted until 2026-11-01 22:23:45.117940655 +0000 UTC m=+65.291749755 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxhl8" (UniqueName: "kubernetes.io/projected/b439b405-8bca-4ce8-980d-a076cb2f11bf-kube-api-access-lxhl8") pod "network-check-target-cfnmp" (UID: "b439b405-8bca-4ce8-980d-a076cb2f11bf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:23:13.156961 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.156930 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-238.us-east-2.compute.internal" event="NodeReady" Nov 01 22:23:13.157090 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.157043 2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 01 22:23:13.200219 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.200188 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-srdnl"] Nov 01 22:23:13.212390 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.212360 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-v2l2j"] Nov 01 22:23:13.212529 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.212486 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-srdnl" Nov 01 22:23:13.216098 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.216070 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fdq7t\"" Nov 01 22:23:13.216098 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.216077 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 22:23:13.216266 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.216127 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 22:23:13.239641 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.239551 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-srdnl"] Nov 01 22:23:13.239783 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.239652 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v2l2j"] Nov 01 22:23:13.239783 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.239676 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v2l2j" Nov 01 22:23:13.316605 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.316571 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 22:23:13.316605 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.316571 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 22:23:13.316792 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.316630 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hhd78\"" Nov 01 22:23:13.316792 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.316670 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 22:23:13.318933 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.318916 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjzrz\" (UniqueName: \"kubernetes.io/projected/b3a3cc56-6cfb-4092-ba39-5c237ad58c77-kube-api-access-qjzrz\") pod \"ingress-canary-v2l2j\" (UID: \"b3a3cc56-6cfb-4092-ba39-5c237ad58c77\") " pod="openshift-ingress-canary/ingress-canary-v2l2j" Nov 01 22:23:13.318972 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.318953 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9af8b8a8-47f6-4576-87db-2f0253903e5b-metrics-tls\") pod \"dns-default-srdnl\" (UID: \"9af8b8a8-47f6-4576-87db-2f0253903e5b\") " pod="openshift-dns/dns-default-srdnl" Nov 01 22:23:13.319007 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.318997 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9af8b8a8-47f6-4576-87db-2f0253903e5b-tmp-dir\") pod \"dns-default-srdnl\" (UID: \"9af8b8a8-47f6-4576-87db-2f0253903e5b\") " pod="openshift-dns/dns-default-srdnl" Nov 01 22:23:13.319079 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.319066 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3a3cc56-6cfb-4092-ba39-5c237ad58c77-cert\") pod \"ingress-canary-v2l2j\" (UID: \"b3a3cc56-6cfb-4092-ba39-5c237ad58c77\") " pod="openshift-ingress-canary/ingress-canary-v2l2j" Nov 01 22:23:13.319117 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.319105 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8wg9\" (UniqueName: \"kubernetes.io/projected/9af8b8a8-47f6-4576-87db-2f0253903e5b-kube-api-access-h8wg9\") pod \"dns-default-srdnl\" (UID: \"9af8b8a8-47f6-4576-87db-2f0253903e5b\") " pod="openshift-dns/dns-default-srdnl" Nov 01 22:23:13.319157 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.319136 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9af8b8a8-47f6-4576-87db-2f0253903e5b-config-volume\") pod \"dns-default-srdnl\" (UID: \"9af8b8a8-47f6-4576-87db-2f0253903e5b\") " pod="openshift-dns/dns-default-srdnl" Nov 01 22:23:13.400796 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.400734 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:23:13.400796 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.400757 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:23:13.400962 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.400945 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:23:13.404645 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.404621 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 22:23:13.404766 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.404693 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 22:23:13.404766 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.404719 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 22:23:13.404866 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.404824 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sw8fs\"" Nov 01 22:23:13.404907 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.404874 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mvk6z\"" Nov 01 22:23:13.404944 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.404936 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sml6h\"" Nov 01 22:23:13.405722 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.405706 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 22:23:13.419945 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.419927 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9af8b8a8-47f6-4576-87db-2f0253903e5b-metrics-tls\") pod \"dns-default-srdnl\" (UID: \"9af8b8a8-47f6-4576-87db-2f0253903e5b\") " pod="openshift-dns/dns-default-srdnl" Nov 01 22:23:13.420034 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.419961 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9af8b8a8-47f6-4576-87db-2f0253903e5b-tmp-dir\") pod \"dns-default-srdnl\" (UID: \"9af8b8a8-47f6-4576-87db-2f0253903e5b\") " pod="openshift-dns/dns-default-srdnl" Nov 01 22:23:13.420034 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.419984 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3a3cc56-6cfb-4092-ba39-5c237ad58c77-cert\") pod \"ingress-canary-v2l2j\" (UID: \"b3a3cc56-6cfb-4092-ba39-5c237ad58c77\") " pod="openshift-ingress-canary/ingress-canary-v2l2j" Nov 01 22:23:13.420034 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.420007 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8wg9\" (UniqueName: \"kubernetes.io/projected/9af8b8a8-47f6-4576-87db-2f0253903e5b-kube-api-access-h8wg9\") pod \"dns-default-srdnl\" (UID: \"9af8b8a8-47f6-4576-87db-2f0253903e5b\") " pod="openshift-dns/dns-default-srdnl" Nov 01 22:23:13.420176 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.420053 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9af8b8a8-47f6-4576-87db-2f0253903e5b-config-volume\") pod \"dns-default-srdnl\" (UID: \"9af8b8a8-47f6-4576-87db-2f0253903e5b\") " pod="openshift-dns/dns-default-srdnl" Nov 01 22:23:13.420176 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:13.420060 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 22:23:13.420176 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.420079 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjzrz\" (UniqueName: \"kubernetes.io/projected/b3a3cc56-6cfb-4092-ba39-5c237ad58c77-kube-api-access-qjzrz\") pod \"ingress-canary-v2l2j\" (UID: \"b3a3cc56-6cfb-4092-ba39-5c237ad58c77\") " pod="openshift-ingress-canary/ingress-canary-v2l2j" Nov 01 22:23:13.420176 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:13.420145 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af8b8a8-47f6-4576-87db-2f0253903e5b-metrics-tls podName:9af8b8a8-47f6-4576-87db-2f0253903e5b nodeName:}" failed. No retries permitted until 2026-11-01 22:23:13.920125361 +0000 UTC m=+34.093934480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9af8b8a8-47f6-4576-87db-2f0253903e5b-metrics-tls") pod "dns-default-srdnl" (UID: "9af8b8a8-47f6-4576-87db-2f0253903e5b") : secret "dns-default-metrics-tls" not found Nov 01 22:23:13.420366 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.420240 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9af8b8a8-47f6-4576-87db-2f0253903e5b-tmp-dir\") pod \"dns-default-srdnl\" (UID: \"9af8b8a8-47f6-4576-87db-2f0253903e5b\") " pod="openshift-dns/dns-default-srdnl" Nov 01 22:23:13.420557 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.420539 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9af8b8a8-47f6-4576-87db-2f0253903e5b-config-volume\") pod \"dns-default-srdnl\" (UID: \"9af8b8a8-47f6-4576-87db-2f0253903e5b\") " pod="openshift-dns/dns-default-srdnl" Nov 01 22:23:13.423487 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.423468 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3a3cc56-6cfb-4092-ba39-5c237ad58c77-cert\") pod \"ingress-canary-v2l2j\" (UID: \"b3a3cc56-6cfb-4092-ba39-5c237ad58c77\") " pod="openshift-ingress-canary/ingress-canary-v2l2j" Nov 01 22:23:13.429367 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.429346 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8wg9\" (UniqueName: \"kubernetes.io/projected/9af8b8a8-47f6-4576-87db-2f0253903e5b-kube-api-access-h8wg9\") pod \"dns-default-srdnl\" (UID: \"9af8b8a8-47f6-4576-87db-2f0253903e5b\") " pod="openshift-dns/dns-default-srdnl" Nov 01 22:23:13.429447 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.429392 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjzrz\" (UniqueName: \"kubernetes.io/projected/b3a3cc56-6cfb-4092-ba39-5c237ad58c77-kube-api-access-qjzrz\") pod \"ingress-canary-v2l2j\" (UID: \"b3a3cc56-6cfb-4092-ba39-5c237ad58c77\") " pod="openshift-ingress-canary/ingress-canary-v2l2j" Nov 01 22:23:13.596348 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.596313 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v2l2j" Nov 01 22:23:13.923074 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.923046 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9af8b8a8-47f6-4576-87db-2f0253903e5b-metrics-tls\") pod \"dns-default-srdnl\" (UID: \"9af8b8a8-47f6-4576-87db-2f0253903e5b\") " pod="openshift-dns/dns-default-srdnl" Nov 01 22:23:13.923207 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:13.923190 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 22:23:13.923265 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:13.923255 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af8b8a8-47f6-4576-87db-2f0253903e5b-metrics-tls podName:9af8b8a8-47f6-4576-87db-2f0253903e5b nodeName:}" failed. No retries permitted until 2026-11-01 22:23:14.923240853 +0000 UTC m=+35.097049953 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9af8b8a8-47f6-4576-87db-2f0253903e5b-metrics-tls") pod "dns-default-srdnl" (UID: "9af8b8a8-47f6-4576-87db-2f0253903e5b") : secret "dns-default-metrics-tls" not found Nov 01 22:23:13.926006 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:13.925970 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v2l2j"] Nov 01 22:23:13.929137 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:23:13.929116 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3a3cc56_6cfb_4092_ba39_5c237ad58c77.slice/crio-c048598f70f7855d3d2d4e4bc8cae60c611b1a3384a3de61ef4b6064fe9f6742 WatchSource:0}: Error finding container c048598f70f7855d3d2d4e4bc8cae60c611b1a3384a3de61ef4b6064fe9f6742: Status 404 returned error can't find the container with id c048598f70f7855d3d2d4e4bc8cae60c611b1a3384a3de61ef4b6064fe9f6742 Nov 01 22:23:14.759558 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:14.759516 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v2l2j" event={"ID":"b3a3cc56-6cfb-4092-ba39-5c237ad58c77","Type":"ContainerStarted","Data":"c048598f70f7855d3d2d4e4bc8cae60c611b1a3384a3de61ef4b6064fe9f6742"} Nov 01 22:23:14.931205 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:14.931172 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9af8b8a8-47f6-4576-87db-2f0253903e5b-metrics-tls\") pod \"dns-default-srdnl\" (UID: \"9af8b8a8-47f6-4576-87db-2f0253903e5b\") " pod="openshift-dns/dns-default-srdnl" Nov 01 22:23:14.931362 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:14.931298 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 22:23:14.931362 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:14.931349 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af8b8a8-47f6-4576-87db-2f0253903e5b-metrics-tls podName:9af8b8a8-47f6-4576-87db-2f0253903e5b nodeName:}" failed. No retries permitted until 2026-11-01 22:23:16.931337929 +0000 UTC m=+37.105147030 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9af8b8a8-47f6-4576-87db-2f0253903e5b-metrics-tls") pod "dns-default-srdnl" (UID: "9af8b8a8-47f6-4576-87db-2f0253903e5b") : secret "dns-default-metrics-tls" not found Nov 01 22:23:16.656092 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.656048 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-sk8f6-sf7bv"] Nov 01 22:23:16.681847 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.681804 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lcqwg"] Nov 01 22:23:16.682001 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.681943 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-sk8f6-sf7bv" Nov 01 22:23:16.688619 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.688592 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 01 22:23:16.688767 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.688653 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-gz5rr\"" Nov 01 22:23:16.689792 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.689767 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 22:23:16.689915 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.689868 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 22:23:16.697071 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.697043 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-sk8f6-sf7bv"] Nov 01 22:23:16.697166 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.697090 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lcqwg"] Nov 01 22:23:16.697239 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.697226 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:16.698152 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.697894 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 01 22:23:16.701531 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.701513 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l7bq7\"" Nov 01 22:23:16.792726 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.792684 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 22:23:16.792726 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.792682 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 22:23:16.845304 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.845259 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb7b37b5-9639-45da-b90e-e08dcfac93f3-crio-socket\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:16.845473 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.845317 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-archives-path\") pod \"periodic-gathering-sk8f6-sf7bv\" (UID: \"3880904a-615b-44ac-9a2d-7b0f7ad31e0c\") " pod="openshift-insights/periodic-gathering-sk8f6-sf7bv" Nov 01 22:23:16.845473 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.845384 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-service-ca-bundle\") pod \"periodic-gathering-sk8f6-sf7bv\" (UID: \"3880904a-615b-44ac-9a2d-7b0f7ad31e0c\") " pod="openshift-insights/periodic-gathering-sk8f6-sf7bv" Nov 01 22:23:16.845473 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.845410 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-trusted-ca-bundle\") pod \"periodic-gathering-sk8f6-sf7bv\" (UID: \"3880904a-615b-44ac-9a2d-7b0f7ad31e0c\") " pod="openshift-insights/periodic-gathering-sk8f6-sf7bv" Nov 01 22:23:16.845473 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.845460 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb7b37b5-9639-45da-b90e-e08dcfac93f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:16.845663 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.845518 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb7b37b5-9639-45da-b90e-e08dcfac93f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:16.845663 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.845583 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb7b37b5-9639-45da-b90e-e08dcfac93f3-data-volume\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:16.845663 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.845648 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ndlv\" (UniqueName: \"kubernetes.io/projected/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-kube-api-access-8ndlv\") pod \"periodic-gathering-sk8f6-sf7bv\" (UID: \"3880904a-615b-44ac-9a2d-7b0f7ad31e0c\") " pod="openshift-insights/periodic-gathering-sk8f6-sf7bv" Nov 01 22:23:16.845794 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.845712 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm2vf\" (UniqueName: \"kubernetes.io/projected/eb7b37b5-9639-45da-b90e-e08dcfac93f3-kube-api-access-vm2vf\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:16.946643 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.946610 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ndlv\" (UniqueName: \"kubernetes.io/projected/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-kube-api-access-8ndlv\") pod \"periodic-gathering-sk8f6-sf7bv\" (UID: \"3880904a-615b-44ac-9a2d-7b0f7ad31e0c\") " pod="openshift-insights/periodic-gathering-sk8f6-sf7bv" Nov 01 22:23:16.946643 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.946651 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9af8b8a8-47f6-4576-87db-2f0253903e5b-metrics-tls\") pod \"dns-default-srdnl\" (UID: \"9af8b8a8-47f6-4576-87db-2f0253903e5b\") " pod="openshift-dns/dns-default-srdnl" Nov 01 22:23:16.946924 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.946670 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm2vf\" (UniqueName: \"kubernetes.io/projected/eb7b37b5-9639-45da-b90e-e08dcfac93f3-kube-api-access-vm2vf\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:16.946924 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.946707 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb7b37b5-9639-45da-b90e-e08dcfac93f3-crio-socket\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:16.946924 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.946726 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-archives-path\") pod \"periodic-gathering-sk8f6-sf7bv\" (UID: \"3880904a-615b-44ac-9a2d-7b0f7ad31e0c\") " pod="openshift-insights/periodic-gathering-sk8f6-sf7bv" Nov 01 22:23:16.946924 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.946743 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-service-ca-bundle\") pod \"periodic-gathering-sk8f6-sf7bv\" (UID: \"3880904a-615b-44ac-9a2d-7b0f7ad31e0c\") " pod="openshift-insights/periodic-gathering-sk8f6-sf7bv" Nov 01 22:23:16.946924 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.946758 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-trusted-ca-bundle\") pod \"periodic-gathering-sk8f6-sf7bv\" (UID: \"3880904a-615b-44ac-9a2d-7b0f7ad31e0c\") " pod="openshift-insights/periodic-gathering-sk8f6-sf7bv" Nov 01 22:23:16.946924 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:16.946770 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 22:23:16.946924 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.946797 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb7b37b5-9639-45da-b90e-e08dcfac93f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:16.946924 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:16.946858 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af8b8a8-47f6-4576-87db-2f0253903e5b-metrics-tls podName:9af8b8a8-47f6-4576-87db-2f0253903e5b nodeName:}" failed. No retries permitted until 2026-11-01 22:23:20.946818465 +0000 UTC m=+41.120627566 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9af8b8a8-47f6-4576-87db-2f0253903e5b-metrics-tls") pod "dns-default-srdnl" (UID: "9af8b8a8-47f6-4576-87db-2f0253903e5b") : secret "dns-default-metrics-tls" not found Nov 01 22:23:16.946924 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.946884 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb7b37b5-9639-45da-b90e-e08dcfac93f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:16.946924 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:16.946894 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 22:23:16.947471 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.946931 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb7b37b5-9639-45da-b90e-e08dcfac93f3-data-volume\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:16.947471 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:16.946947 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb7b37b5-9639-45da-b90e-e08dcfac93f3-kube-rbac-proxy-cm podName:eb7b37b5-9639-45da-b90e-e08dcfac93f3 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:17.44693299 +0000 UTC m=+37.620742111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb7b37b5-9639-45da-b90e-e08dcfac93f3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lcqwg" (UID: "eb7b37b5-9639-45da-b90e-e08dcfac93f3") : configmap "kube-rbac-proxy" not found Nov 01 22:23:16.947471 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.946986 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb7b37b5-9639-45da-b90e-e08dcfac93f3-crio-socket\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:16.947471 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:16.947306 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 22:23:16.947471 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:16.947354 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb7b37b5-9639-45da-b90e-e08dcfac93f3-insights-runtime-extractor-tls podName:eb7b37b5-9639-45da-b90e-e08dcfac93f3 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:17.447338988 +0000 UTC m=+37.621148091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb7b37b5-9639-45da-b90e-e08dcfac93f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lcqwg" (UID: "eb7b37b5-9639-45da-b90e-e08dcfac93f3") : secret "insights-runtime-extractor-tls" not found Nov 01 22:23:16.947471 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.947455 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb7b37b5-9639-45da-b90e-e08dcfac93f3-data-volume\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:16.947773 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.947498 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-archives-path\") pod \"periodic-gathering-sk8f6-sf7bv\" (UID: \"3880904a-615b-44ac-9a2d-7b0f7ad31e0c\") " pod="openshift-insights/periodic-gathering-sk8f6-sf7bv" Nov 01 22:23:16.947773 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.947607 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-service-ca-bundle\") pod \"periodic-gathering-sk8f6-sf7bv\" (UID: \"3880904a-615b-44ac-9a2d-7b0f7ad31e0c\") " pod="openshift-insights/periodic-gathering-sk8f6-sf7bv" Nov 01 22:23:16.947898 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.947861 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-trusted-ca-bundle\") pod \"periodic-gathering-sk8f6-sf7bv\" (UID: \"3880904a-615b-44ac-9a2d-7b0f7ad31e0c\") " pod="openshift-insights/periodic-gathering-sk8f6-sf7bv" Nov 01 22:23:16.955291 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.955264 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm2vf\" (UniqueName: \"kubernetes.io/projected/eb7b37b5-9639-45da-b90e-e08dcfac93f3-kube-api-access-vm2vf\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:16.955399 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:16.955381 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ndlv\" (UniqueName: \"kubernetes.io/projected/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-kube-api-access-8ndlv\") pod \"periodic-gathering-sk8f6-sf7bv\" (UID: \"3880904a-615b-44ac-9a2d-7b0f7ad31e0c\") " pod="openshift-insights/periodic-gathering-sk8f6-sf7bv" Nov 01 22:23:17.011617 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:17.011582 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-sk8f6-sf7bv" Nov 01 22:23:17.308002 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:17.307928 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-sk8f6-sf7bv"] Nov 01 22:23:17.393159 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:23:17.393130 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3880904a_615b_44ac_9a2d_7b0f7ad31e0c.slice/crio-7385d8d2150889036d38eaa7fc4e655bf36d9fb0ee541e0edca952a91e2a79c7 WatchSource:0}: Error finding container 7385d8d2150889036d38eaa7fc4e655bf36d9fb0ee541e0edca952a91e2a79c7: Status 404 returned error can't find the container with id 7385d8d2150889036d38eaa7fc4e655bf36d9fb0ee541e0edca952a91e2a79c7 Nov 01 22:23:17.452760 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:17.452737 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb7b37b5-9639-45da-b90e-e08dcfac93f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:17.452866 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:17.452766 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb7b37b5-9639-45da-b90e-e08dcfac93f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:17.452908 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:17.452864 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 22:23:17.452908 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:17.452876 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 22:23:17.453011 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:17.452924 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb7b37b5-9639-45da-b90e-e08dcfac93f3-kube-rbac-proxy-cm podName:eb7b37b5-9639-45da-b90e-e08dcfac93f3 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:18.452909943 +0000 UTC m=+38.626719043 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb7b37b5-9639-45da-b90e-e08dcfac93f3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lcqwg" (UID: "eb7b37b5-9639-45da-b90e-e08dcfac93f3") : configmap "kube-rbac-proxy" not found Nov 01 22:23:17.453011 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:17.452937 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb7b37b5-9639-45da-b90e-e08dcfac93f3-insights-runtime-extractor-tls podName:eb7b37b5-9639-45da-b90e-e08dcfac93f3 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:18.452931358 +0000 UTC m=+38.626740458 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb7b37b5-9639-45da-b90e-e08dcfac93f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lcqwg" (UID: "eb7b37b5-9639-45da-b90e-e08dcfac93f3") : secret "insights-runtime-extractor-tls" not found Nov 01 22:23:17.768074 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:17.768029 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v2l2j" event={"ID":"b3a3cc56-6cfb-4092-ba39-5c237ad58c77","Type":"ContainerStarted","Data":"6a697a3a7fc2edeb3fc424f3815319871c28020ef05fee59759f7a397a8eeccc"} Nov 01 22:23:17.769419 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:17.769350 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-sk8f6-sf7bv" event={"ID":"3880904a-615b-44ac-9a2d-7b0f7ad31e0c","Type":"ContainerStarted","Data":"7385d8d2150889036d38eaa7fc4e655bf36d9fb0ee541e0edca952a91e2a79c7"} Nov 01 22:23:17.783609 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:17.783200 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-v2l2j" podStartSLOduration=1.281338522 podStartE2EDuration="4.783185643s" totalImagesPullingTime="3.501847121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:17.782996559 +0000 UTC m=+37.956805681" watchObservedRunningTime="2026-11-01 22:23:17.783185643 +0000 UTC m=+37.956994765" Nov 01 22:23:18.460780 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:18.460746 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb7b37b5-9639-45da-b90e-e08dcfac93f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:18.460780 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:18.460782 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb7b37b5-9639-45da-b90e-e08dcfac93f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:18.461114 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:18.460891 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 22:23:18.461114 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:18.460961 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb7b37b5-9639-45da-b90e-e08dcfac93f3-kube-rbac-proxy-cm podName:eb7b37b5-9639-45da-b90e-e08dcfac93f3 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:20.460942824 +0000 UTC m=+40.634751946 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb7b37b5-9639-45da-b90e-e08dcfac93f3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lcqwg" (UID: "eb7b37b5-9639-45da-b90e-e08dcfac93f3") : configmap "kube-rbac-proxy" not found Nov 01 22:23:18.461114 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:18.460991 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 22:23:18.461114 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:18.461067 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb7b37b5-9639-45da-b90e-e08dcfac93f3-insights-runtime-extractor-tls podName:eb7b37b5-9639-45da-b90e-e08dcfac93f3 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:20.46105008 +0000 UTC m=+40.634859185 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb7b37b5-9639-45da-b90e-e08dcfac93f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lcqwg" (UID: "eb7b37b5-9639-45da-b90e-e08dcfac93f3") : secret "insights-runtime-extractor-tls" not found Nov 01 22:23:18.601641 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:18.601608 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-ss7dr"] Nov 01 22:23:18.620230 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:18.620202 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-ss7dr"] Nov 01 22:23:18.620385 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:18.620310 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-ss7dr" Nov 01 22:23:18.623463 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:18.623436 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 01 22:23:18.623582 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:18.623566 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 01 22:23:18.624817 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:18.624782 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 01 22:23:18.624984 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:18.624968 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-5lf4f\"" Nov 01 22:23:18.625070 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:18.624993 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 01 22:23:18.764049 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:18.763967 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3eeb8650-24f6-4e09-95be-794fe6a925bc-signing-cabundle\") pod \"service-ca-5fffc97bdf-ss7dr\" (UID: \"3eeb8650-24f6-4e09-95be-794fe6a925bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ss7dr" Nov 01 22:23:18.764203 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:18.764082 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdvfj\" (UniqueName: \"kubernetes.io/projected/3eeb8650-24f6-4e09-95be-794fe6a925bc-kube-api-access-tdvfj\") pod \"service-ca-5fffc97bdf-ss7dr\" (UID: \"3eeb8650-24f6-4e09-95be-794fe6a925bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ss7dr" Nov 01 22:23:18.764203 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:18.764115 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3eeb8650-24f6-4e09-95be-794fe6a925bc-signing-key\") pod \"service-ca-5fffc97bdf-ss7dr\" (UID: \"3eeb8650-24f6-4e09-95be-794fe6a925bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ss7dr" Nov 01 22:23:18.864504 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:18.864474 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3eeb8650-24f6-4e09-95be-794fe6a925bc-signing-cabundle\") pod \"service-ca-5fffc97bdf-ss7dr\" (UID: \"3eeb8650-24f6-4e09-95be-794fe6a925bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ss7dr" Nov 01 22:23:18.865108 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:18.864546 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdvfj\" (UniqueName: \"kubernetes.io/projected/3eeb8650-24f6-4e09-95be-794fe6a925bc-kube-api-access-tdvfj\") pod \"service-ca-5fffc97bdf-ss7dr\" (UID: \"3eeb8650-24f6-4e09-95be-794fe6a925bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ss7dr" Nov 01 22:23:18.865108 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:18.864569 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3eeb8650-24f6-4e09-95be-794fe6a925bc-signing-key\") pod \"service-ca-5fffc97bdf-ss7dr\" (UID: \"3eeb8650-24f6-4e09-95be-794fe6a925bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ss7dr" Nov 01 22:23:18.865177 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:18.865146 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3eeb8650-24f6-4e09-95be-794fe6a925bc-signing-cabundle\") pod \"service-ca-5fffc97bdf-ss7dr\" (UID: \"3eeb8650-24f6-4e09-95be-794fe6a925bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ss7dr" Nov 01 22:23:18.867762 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:18.867739 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3eeb8650-24f6-4e09-95be-794fe6a925bc-signing-key\") pod \"service-ca-5fffc97bdf-ss7dr\" (UID: \"3eeb8650-24f6-4e09-95be-794fe6a925bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ss7dr" Nov 01 22:23:18.872878 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:18.872859 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdvfj\" (UniqueName: \"kubernetes.io/projected/3eeb8650-24f6-4e09-95be-794fe6a925bc-kube-api-access-tdvfj\") pod \"service-ca-5fffc97bdf-ss7dr\" (UID: \"3eeb8650-24f6-4e09-95be-794fe6a925bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ss7dr" Nov 01 22:23:18.946967 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:18.946935 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-ss7dr" Nov 01 22:23:19.242192 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:19.241954 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-ss7dr"] Nov 01 22:23:19.245593 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:23:19.245567 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eeb8650_24f6_4e09_95be_794fe6a925bc.slice/crio-5cf72dade1beccabe52d7cdfe6ddac10ac191dbcd514ce18ffb70d597769e61c WatchSource:0}: Error finding container 5cf72dade1beccabe52d7cdfe6ddac10ac191dbcd514ce18ffb70d597769e61c: Status 404 returned error can't find the container with id 5cf72dade1beccabe52d7cdfe6ddac10ac191dbcd514ce18ffb70d597769e61c Nov 01 22:23:19.435265 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:19.435239 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-9s6vz"] Nov 01 22:23:19.458503 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:19.458473 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-9s6vz"] Nov 01 22:23:19.458668 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:19.458616 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9s6vz" Nov 01 22:23:19.464255 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:19.464219 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-d6jqb\"" Nov 01 22:23:19.471184 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:19.471144 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 01 22:23:19.471305 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:19.471215 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 01 22:23:19.572070 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:19.572023 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64mk6\" (UniqueName: \"kubernetes.io/projected/54ac1aee-48aa-4f54-ae3c-9d5d4e377e12-kube-api-access-64mk6\") pod \"migrator-66df5d496b-9s6vz\" (UID: \"54ac1aee-48aa-4f54-ae3c-9d5d4e377e12\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9s6vz" Nov 01 22:23:19.673553 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:19.673513 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64mk6\" (UniqueName: \"kubernetes.io/projected/54ac1aee-48aa-4f54-ae3c-9d5d4e377e12-kube-api-access-64mk6\") pod \"migrator-66df5d496b-9s6vz\" (UID: \"54ac1aee-48aa-4f54-ae3c-9d5d4e377e12\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9s6vz" Nov 01 22:23:19.683400 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:19.683364 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64mk6\" (UniqueName: \"kubernetes.io/projected/54ac1aee-48aa-4f54-ae3c-9d5d4e377e12-kube-api-access-64mk6\") pod \"migrator-66df5d496b-9s6vz\" (UID: \"54ac1aee-48aa-4f54-ae3c-9d5d4e377e12\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9s6vz" Nov 01 22:23:19.776652 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:19.776586 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-ss7dr" event={"ID":"3eeb8650-24f6-4e09-95be-794fe6a925bc","Type":"ContainerStarted","Data":"5cf72dade1beccabe52d7cdfe6ddac10ac191dbcd514ce18ffb70d597769e61c"} Nov 01 22:23:19.786745 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:19.786719 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9s6vz" Nov 01 22:23:20.113755 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:20.113686 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-9s6vz"] Nov 01 22:23:20.217263 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:23:20.217220 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ac1aee_48aa_4f54_ae3c_9d5d4e377e12.slice/crio-46c075f064de3dcfb4faafb0bd55cdc02e69e0867346a8cd3c44a165829abd38 WatchSource:0}: Error finding container 46c075f064de3dcfb4faafb0bd55cdc02e69e0867346a8cd3c44a165829abd38: Status 404 returned error can't find the container with id 46c075f064de3dcfb4faafb0bd55cdc02e69e0867346a8cd3c44a165829abd38 Nov 01 22:23:20.480750 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:20.480714 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb7b37b5-9639-45da-b90e-e08dcfac93f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:20.480998 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:20.480756 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb7b37b5-9639-45da-b90e-e08dcfac93f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:20.480998 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:20.480924 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 22:23:20.480998 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:20.480986 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb7b37b5-9639-45da-b90e-e08dcfac93f3-insights-runtime-extractor-tls podName:eb7b37b5-9639-45da-b90e-e08dcfac93f3 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:24.480971282 +0000 UTC m=+44.654780384 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb7b37b5-9639-45da-b90e-e08dcfac93f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lcqwg" (UID: "eb7b37b5-9639-45da-b90e-e08dcfac93f3") : secret "insights-runtime-extractor-tls" not found Nov 01 22:23:20.481315 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:20.481283 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb7b37b5-9639-45da-b90e-e08dcfac93f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:20.780784 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:20.780693 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-sk8f6-sf7bv" event={"ID":"3880904a-615b-44ac-9a2d-7b0f7ad31e0c","Type":"ContainerStarted","Data":"fe116260e63ccb4c5c548d63bebf6b667b07d5f9b4ce7deaa367b417a7fcd04d"} Nov 01 22:23:20.781791 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:20.781769 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9s6vz" event={"ID":"54ac1aee-48aa-4f54-ae3c-9d5d4e377e12","Type":"ContainerStarted","Data":"46c075f064de3dcfb4faafb0bd55cdc02e69e0867346a8cd3c44a165829abd38"} Nov 01 22:23:20.797128 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:20.797069 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-sk8f6-sf7bv" podStartSLOduration=1.967308692 podStartE2EDuration="4.797054616s" totalImagesPullingTime="2.829745924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:20.796645263 +0000 UTC m=+40.970454386" watchObservedRunningTime="2026-11-01 22:23:20.797054616 +0000 UTC m=+40.970863737" Nov 01 22:23:20.985333 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:20.985296 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9af8b8a8-47f6-4576-87db-2f0253903e5b-metrics-tls\") pod \"dns-default-srdnl\" (UID: \"9af8b8a8-47f6-4576-87db-2f0253903e5b\") " pod="openshift-dns/dns-default-srdnl" Nov 01 22:23:20.985530 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:20.985472 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 22:23:20.985594 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:20.985555 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af8b8a8-47f6-4576-87db-2f0253903e5b-metrics-tls podName:9af8b8a8-47f6-4576-87db-2f0253903e5b nodeName:}" failed. No retries permitted until 2026-11-01 22:23:28.985532322 +0000 UTC m=+49.159341435 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9af8b8a8-47f6-4576-87db-2f0253903e5b-metrics-tls") pod "dns-default-srdnl" (UID: "9af8b8a8-47f6-4576-87db-2f0253903e5b") : secret "dns-default-metrics-tls" not found Nov 01 22:23:23.071759 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:23.071730 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x4k8b_f71460db-658f-45ed-980f-da65509e2bc5/dns-node-resolver/0.log" Nov 01 22:23:23.671859 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:23.671812 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rkf2j_682b0c5d-fa6f-4a17-8b72-a1f224cf5fec/node-ca/0.log" Nov 01 22:23:24.472089 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:24.472001 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v2l2j_b3a3cc56-6cfb-4092-ba39-5c237ad58c77/serve-healthcheck-canary/0.log" Nov 01 22:23:24.512212 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:24.512174 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb7b37b5-9639-45da-b90e-e08dcfac93f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:24.512380 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:24.512360 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 22:23:24.512473 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:24.512441 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb7b37b5-9639-45da-b90e-e08dcfac93f3-insights-runtime-extractor-tls podName:eb7b37b5-9639-45da-b90e-e08dcfac93f3 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:32.512419907 +0000 UTC m=+52.686229023 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb7b37b5-9639-45da-b90e-e08dcfac93f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lcqwg" (UID: "eb7b37b5-9639-45da-b90e-e08dcfac93f3") : secret "insights-runtime-extractor-tls" not found Nov 01 22:23:24.793866 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:24.793771 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9s6vz" event={"ID":"54ac1aee-48aa-4f54-ae3c-9d5d4e377e12","Type":"ContainerStarted","Data":"4908c457dc368b8faa6c27d5a6fef48a2b20bf7dcfb4463213057a4d9d8747a9"} Nov 01 22:23:24.809840 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:24.809785 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9s6vz" podStartSLOduration=2.284899341 podStartE2EDuration="5.809766975s" totalImagesPullingTime="3.524867634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:24.809291671 +0000 UTC m=+44.983100793" watchObservedRunningTime="2026-11-01 22:23:24.809766975 +0000 UTC m=+44.983576096" Nov 01 22:23:24.872687 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:24.872659 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-9s6vz_54ac1aee-48aa-4f54-ae3c-9d5d4e377e12/migrator/0.log" Nov 01 22:23:25.797698 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:25.797662 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-ss7dr" event={"ID":"3eeb8650-24f6-4e09-95be-794fe6a925bc","Type":"ContainerStarted","Data":"e6b5d61e5488638fa5d0f51a2b3cf7e5c0dd37e5f66ebe9916905e7eefed9586"} Nov 01 22:23:25.814896 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:25.814813 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-ss7dr" podStartSLOduration=1.988558029 podStartE2EDuration="7.814799639s" totalImagesPullingTime="5.82624161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:25.814565753 +0000 UTC m=+45.988374874" watchObservedRunningTime="2026-11-01 22:23:25.814799639 +0000 UTC m=+45.988608761" Nov 01 22:23:29.044744 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:29.044711 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9af8b8a8-47f6-4576-87db-2f0253903e5b-metrics-tls\") pod \"dns-default-srdnl\" (UID: \"9af8b8a8-47f6-4576-87db-2f0253903e5b\") " pod="openshift-dns/dns-default-srdnl" Nov 01 22:23:29.045217 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:29.044904 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 22:23:29.045217 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:29.044971 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af8b8a8-47f6-4576-87db-2f0253903e5b-metrics-tls podName:9af8b8a8-47f6-4576-87db-2f0253903e5b nodeName:}" failed. No retries permitted until 2026-11-01 22:23:45.04495473 +0000 UTC m=+65.218763849 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9af8b8a8-47f6-4576-87db-2f0253903e5b-metrics-tls") pod "dns-default-srdnl" (UID: "9af8b8a8-47f6-4576-87db-2f0253903e5b") : secret "dns-default-metrics-tls" not found Nov 01 22:23:32.568946 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:32.568892 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb7b37b5-9639-45da-b90e-e08dcfac93f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:32.569392 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:32.569050 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 22:23:32.569392 ip-10-0-142-238 kubenswrapper[2576]: E1101 22:23:32.569121 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb7b37b5-9639-45da-b90e-e08dcfac93f3-insights-runtime-extractor-tls podName:eb7b37b5-9639-45da-b90e-e08dcfac93f3 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:48.569105355 +0000 UTC m=+68.742914468 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb7b37b5-9639-45da-b90e-e08dcfac93f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lcqwg" (UID: "eb7b37b5-9639-45da-b90e-e08dcfac93f3") : secret "insights-runtime-extractor-tls" not found Nov 01 22:23:37.754191 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:37.754158 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:45.057691 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:45.057653 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9af8b8a8-47f6-4576-87db-2f0253903e5b-metrics-tls\") pod \"dns-default-srdnl\" (UID: \"9af8b8a8-47f6-4576-87db-2f0253903e5b\") " pod="openshift-dns/dns-default-srdnl" Nov 01 22:23:45.058118 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:45.057720 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa603801-6c30-4cf1-ace8-30c8723e4a1e-original-pull-secret\") pod \"global-pull-secret-syncer-ttjsz\" (UID: \"fa603801-6c30-4cf1-ace8-30c8723e4a1e\") " pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:23:45.060700 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:45.060672 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 22:23:45.061529 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:45.061511 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9af8b8a8-47f6-4576-87db-2f0253903e5b-metrics-tls\") pod \"dns-default-srdnl\" (UID: \"9af8b8a8-47f6-4576-87db-2f0253903e5b\") " pod="openshift-dns/dns-default-srdnl" Nov 01 22:23:45.070427 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:45.070403 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa603801-6c30-4cf1-ace8-30c8723e4a1e-original-pull-secret\") pod \"global-pull-secret-syncer-ttjsz\" (UID: \"fa603801-6c30-4cf1-ace8-30c8723e4a1e\") " pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:23:45.076759 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:45.076738 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fdq7t\"" Nov 01 22:23:45.084882 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:45.084867 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-srdnl" Nov 01 22:23:45.158515 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:45.158477 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc3de28d-1139-4677-81db-a114c8867933-metrics-certs\") pod \"network-metrics-daemon-xvgrl\" (UID: \"cc3de28d-1139-4677-81db-a114c8867933\") " pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:23:45.158642 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:45.158520 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxhl8\" (UniqueName: \"kubernetes.io/projected/b439b405-8bca-4ce8-980d-a076cb2f11bf-kube-api-access-lxhl8\") pod \"network-check-target-cfnmp\" (UID: \"b439b405-8bca-4ce8-980d-a076cb2f11bf\") " pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:23:45.161265 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:45.161247 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 22:23:45.161355 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:45.161288 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 22:23:45.171508 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:45.171491 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 22:23:45.171594 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:45.171565 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc3de28d-1139-4677-81db-a114c8867933-metrics-certs\") pod \"network-metrics-daemon-xvgrl\" (UID: \"cc3de28d-1139-4677-81db-a114c8867933\") " pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:23:45.181776 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:45.181752 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxhl8\" (UniqueName: \"kubernetes.io/projected/b439b405-8bca-4ce8-980d-a076cb2f11bf-kube-api-access-lxhl8\") pod \"network-check-target-cfnmp\" (UID: \"b439b405-8bca-4ce8-980d-a076cb2f11bf\") " pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:23:45.236504 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:45.236476 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mvk6z\"" Nov 01 22:23:45.244565 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:45.244539 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:23:45.268356 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:45.268333 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sw8fs\"" Nov 01 22:23:45.276574 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:45.276555 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:23:45.304745 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:45.304720 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sml6h\"" Nov 01 22:23:45.312545 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:45.312441 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:23:45.357753 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:45.357726 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-srdnl"] Nov 01 22:23:45.361601 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:23:45.361572 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af8b8a8_47f6_4576_87db_2f0253903e5b.slice/crio-f893907a88ffd76773a15165a7030c7870cbd584687312cc74d847dc10358a55 WatchSource:0}: Error finding container f893907a88ffd76773a15165a7030c7870cbd584687312cc74d847dc10358a55: Status 404 returned error can't find the container with id f893907a88ffd76773a15165a7030c7870cbd584687312cc74d847dc10358a55 Nov 01 22:23:45.517549 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:45.517514 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ttjsz"] Nov 01 22:23:45.520168 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:23:45.520125 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa603801_6c30_4cf1_ace8_30c8723e4a1e.slice/crio-ca29d6a9e0c4f4af79bc2de4727b64eabf75000f0040f0564f732c7ee247157a WatchSource:0}: Error finding container ca29d6a9e0c4f4af79bc2de4727b64eabf75000f0040f0564f732c7ee247157a: Status 404 returned error can't find the container with id ca29d6a9e0c4f4af79bc2de4727b64eabf75000f0040f0564f732c7ee247157a Nov 01 22:23:45.550448 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:45.550419 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cfnmp"] Nov 01 22:23:45.553815 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:23:45.553788 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb439b405_8bca_4ce8_980d_a076cb2f11bf.slice/crio-25bfba8acd1e8775dfcc86ff3f4e202898c47daba1a52956e44466ce2d833b47 WatchSource:0}: Error finding container 25bfba8acd1e8775dfcc86ff3f4e202898c47daba1a52956e44466ce2d833b47: Status 404 returned error can't find the container with id 25bfba8acd1e8775dfcc86ff3f4e202898c47daba1a52956e44466ce2d833b47 Nov 01 22:23:45.589572 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:45.589507 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xvgrl"] Nov 01 22:23:45.592141 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:23:45.592114 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc3de28d_1139_4677_81db_a114c8867933.slice/crio-af19768f02c54b9ae643154322ededc99c4bbbb36ffce3831f3930c16bc8fdc0 WatchSource:0}: Error finding container af19768f02c54b9ae643154322ededc99c4bbbb36ffce3831f3930c16bc8fdc0: Status 404 returned error can't find the container with id af19768f02c54b9ae643154322ededc99c4bbbb36ffce3831f3930c16bc8fdc0 Nov 01 22:23:45.846923 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:45.846819 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cfnmp" event={"ID":"b439b405-8bca-4ce8-980d-a076cb2f11bf","Type":"ContainerStarted","Data":"25bfba8acd1e8775dfcc86ff3f4e202898c47daba1a52956e44466ce2d833b47"} Nov 01 22:23:45.848056 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:45.848030 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-srdnl" event={"ID":"9af8b8a8-47f6-4576-87db-2f0253903e5b","Type":"ContainerStarted","Data":"f893907a88ffd76773a15165a7030c7870cbd584687312cc74d847dc10358a55"} Nov 01 22:23:45.849245 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:45.849220 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ttjsz" event={"ID":"fa603801-6c30-4cf1-ace8-30c8723e4a1e","Type":"ContainerStarted","Data":"ca29d6a9e0c4f4af79bc2de4727b64eabf75000f0040f0564f732c7ee247157a"} Nov 01 22:23:45.850273 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:45.850254 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xvgrl" event={"ID":"cc3de28d-1139-4677-81db-a114c8867933","Type":"ContainerStarted","Data":"af19768f02c54b9ae643154322ededc99c4bbbb36ffce3831f3930c16bc8fdc0"} Nov 01 22:23:47.861064 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:47.861020 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xvgrl" event={"ID":"cc3de28d-1139-4677-81db-a114c8867933","Type":"ContainerStarted","Data":"8300a1b386be23dde0e62d458bd1da971c4efdcd85cc447dd57ff22122531d09"} Nov 01 22:23:47.861611 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:47.861088 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xvgrl" event={"ID":"cc3de28d-1139-4677-81db-a114c8867933","Type":"ContainerStarted","Data":"81d04160cf62842ca10238440ea17165c9902cb80fa3d5842174c03ea4872b32"} Nov 01 22:23:47.878514 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:47.878456 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xvgrl" podStartSLOduration=65.285019088 podStartE2EDuration="1m6.878438048s" totalImagesPullingTime="1.59341896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:47.877596656 +0000 UTC m=+68.051405777" watchObservedRunningTime="2026-11-01 22:23:47.878438048 +0000 UTC m=+68.052247170" Nov 01 22:23:48.584863 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:48.584513 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb7b37b5-9639-45da-b90e-e08dcfac93f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:48.587289 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:48.587262 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb7b37b5-9639-45da-b90e-e08dcfac93f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:48.866769 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:48.866679 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-srdnl" event={"ID":"9af8b8a8-47f6-4576-87db-2f0253903e5b","Type":"ContainerStarted","Data":"e17dd3ca79fec3f26be36accbe8c185c912ef202156bbc5f26b22b343b0026cf"} Nov 01 22:23:48.866769 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:48.866740 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-srdnl" event={"ID":"9af8b8a8-47f6-4576-87db-2f0253903e5b","Type":"ContainerStarted","Data":"43c353c9e8fc992a2aae8d8c1de12f495b040da63a66628f27a40d3cab3f73d2"} Nov 01 22:23:48.867270 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:48.866948 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l7bq7\"" Nov 01 22:23:48.867270 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:48.867064 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-srdnl" Nov 01 22:23:48.875420 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:48.875385 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:48.883895 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:48.883849 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-srdnl" podStartSLOduration=32.724376416 podStartE2EDuration="35.88381179s" totalImagesPullingTime="3.159435374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:48.88279003 +0000 UTC m=+69.056599153" watchObservedRunningTime="2026-11-01 22:23:48.88381179 +0000 UTC m=+69.057620912" Nov 01 22:23:49.159241 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:49.159165 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lcqwg"] Nov 01 22:23:49.162631 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:23:49.162588 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb7b37b5_9639_45da_b90e_e08dcfac93f3.slice/crio-9e5f098e5bd56bb34954af334e7fa0786eb3399bd7c3362f0506d074be9a8579 WatchSource:0}: Error finding container 9e5f098e5bd56bb34954af334e7fa0786eb3399bd7c3362f0506d074be9a8579: Status 404 returned error can't find the container with id 9e5f098e5bd56bb34954af334e7fa0786eb3399bd7c3362f0506d074be9a8579 Nov 01 22:23:49.253485 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:49.253442 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-sxspm"] Nov 01 22:23:49.258382 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:49.257092 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-sxspm" Nov 01 22:23:49.265880 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:49.265845 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-sxspm"] Nov 01 22:23:49.269597 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:49.269550 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 22:23:49.269942 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:49.269906 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 01 22:23:49.270317 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:49.270277 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 22:23:49.270317 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:49.270292 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 22:23:49.270456 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:49.270318 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 01 22:23:49.273035 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:49.273014 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-rrpdg\"" Nov 01 22:23:49.288279 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:49.288253 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-sxspm\" (UID: \"e296d10d-5064-4f8f-85f0-e6d1b20f1ab8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-sxspm" Nov 01 22:23:49.288418 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:49.288287 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-sxspm\" (UID: \"e296d10d-5064-4f8f-85f0-e6d1b20f1ab8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-sxspm" Nov 01 22:23:49.288418 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:49.288321 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-sxspm\" (UID: \"e296d10d-5064-4f8f-85f0-e6d1b20f1ab8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-sxspm" Nov 01 22:23:49.288418 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:49.288338 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwr44\" (UniqueName: \"kubernetes.io/projected/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8-kube-api-access-gwr44\") pod \"prometheus-operator-9cd89b74c-sxspm\" (UID: \"e296d10d-5064-4f8f-85f0-e6d1b20f1ab8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-sxspm" Nov 01 22:23:49.389517 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:49.389481 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-sxspm\" (UID: \"e296d10d-5064-4f8f-85f0-e6d1b20f1ab8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-sxspm" Nov 01 22:23:49.389517 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:49.389524 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwr44\" (UniqueName: \"kubernetes.io/projected/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8-kube-api-access-gwr44\") pod \"prometheus-operator-9cd89b74c-sxspm\" (UID: \"e296d10d-5064-4f8f-85f0-e6d1b20f1ab8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-sxspm" Nov 01 22:23:49.389731 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:49.389550 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-sxspm\" (UID: \"e296d10d-5064-4f8f-85f0-e6d1b20f1ab8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-sxspm" Nov 01 22:23:49.389731 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:49.389572 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-sxspm\" (UID: \"e296d10d-5064-4f8f-85f0-e6d1b20f1ab8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-sxspm" Nov 01 22:23:49.390589 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:49.390563 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-sxspm\" (UID: \"e296d10d-5064-4f8f-85f0-e6d1b20f1ab8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-sxspm" Nov 01 22:23:49.392574 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:49.392541 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-sxspm\" (UID: \"e296d10d-5064-4f8f-85f0-e6d1b20f1ab8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-sxspm" Nov 01 22:23:49.392759 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:49.392741 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-sxspm\" (UID: \"e296d10d-5064-4f8f-85f0-e6d1b20f1ab8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-sxspm" Nov 01 22:23:49.398777 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:49.398757 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwr44\" (UniqueName: \"kubernetes.io/projected/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8-kube-api-access-gwr44\") pod \"prometheus-operator-9cd89b74c-sxspm\" (UID: \"e296d10d-5064-4f8f-85f0-e6d1b20f1ab8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-sxspm" Nov 01 22:23:49.578617 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:49.578590 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-sxspm" Nov 01 22:23:49.858469 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:49.858400 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-sxspm"] Nov 01 22:23:49.863580 ip-10-0-142-238 kubenswrapper[2576]: W1101 22:23:49.862583 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode296d10d_5064_4f8f_85f0_e6d1b20f1ab8.slice/crio-01c5e91dacb94194c3d671b9a66d0421dc6640b674a9d907d1ebfb40ce4e8632 WatchSource:0}: Error finding container 01c5e91dacb94194c3d671b9a66d0421dc6640b674a9d907d1ebfb40ce4e8632: Status 404 returned error can't find the container with id 01c5e91dacb94194c3d671b9a66d0421dc6640b674a9d907d1ebfb40ce4e8632 Nov 01 22:23:49.871486 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:49.871376 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lcqwg" event={"ID":"eb7b37b5-9639-45da-b90e-e08dcfac93f3","Type":"ContainerStarted","Data":"cf34d3b44e405c644d4a900e09eacbe564416023bc0514d61ed2850785b21428"} Nov 01 22:23:49.871486 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:49.871445 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lcqwg" event={"ID":"eb7b37b5-9639-45da-b90e-e08dcfac93f3","Type":"ContainerStarted","Data":"9e5f098e5bd56bb34954af334e7fa0786eb3399bd7c3362f0506d074be9a8579"} Nov 01 22:23:49.873028 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:49.872998 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cfnmp" event={"ID":"b439b405-8bca-4ce8-980d-a076cb2f11bf","Type":"ContainerStarted","Data":"3218a613d3a46566387649237966947b5bb3c81c6a65dfc19d6f60bb508eb0c7"} Nov 01 22:23:49.873127 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:49.873073 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:23:49.874427 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:49.874398 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-sxspm" event={"ID":"e296d10d-5064-4f8f-85f0-e6d1b20f1ab8","Type":"ContainerStarted","Data":"01c5e91dacb94194c3d671b9a66d0421dc6640b674a9d907d1ebfb40ce4e8632"} Nov 01 22:23:49.890544 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:49.890484 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cfnmp" podStartSLOduration=64.903258888 podStartE2EDuration="1m8.890470016s" totalImagesPullingTime="3.987211128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:49.889850284 +0000 UTC m=+70.063659398" watchObservedRunningTime="2026-11-01 22:23:49.890470016 +0000 UTC m=+70.064279137" Nov 01 22:23:50.880973 ip-10-0-142-238 kubenswrapper[2576]: I1101 22:23:50.880933 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lcqwg" event={"ID":"eb7b37b5-9639-45da-b90e-e08dcfac93f3","Type":"ContainerStarted","Data":"28ad8f75efbaf9ee2be9f46c755d2b4c757ee2c373f67247bc61f13c90d31b3f"} Nov 01 22:23:51.507407 ip-10-0-142-238 systemd[1]: Stopping Kubernetes Kubelet... Nov 01 22:23:51.520291 ip-10-0-142-238 systemd[1]: kubelet.service: Deactivated successfully. Nov 01 22:23:51.520487 ip-10-0-142-238 systemd[1]: Stopped Kubernetes Kubelet. Nov 01 22:23:51.520692 ip-10-0-142-238 systemd[1]: kubelet.service: Consumed 3.397s CPU time. Nov 01 22:23:51.545469 ip-10-0-142-238 systemd[1]: Starting Kubernetes Kubelet... Nov 01 22:23:51.631371 ip-10-0-142-238 kubenswrapper[5011]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 22:23:51.631371 ip-10-0-142-238 kubenswrapper[5011]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 22:23:51.631371 ip-10-0-142-238 kubenswrapper[5011]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 22:23:51.631371 ip-10-0-142-238 kubenswrapper[5011]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 22:23:51.633749 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633655 5011 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:51.633749 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633746 5011 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:51.633749 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633751 5011 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:51.633749 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633755 5011 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:51.634027 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633759 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:51.634027 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633764 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:51.634027 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633768 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:51.634027 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633771 5011 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:51.634027 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633775 5011 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:51.634027 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633779 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:51.634027 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633782 5011 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:51.634027 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633786 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:51.634027 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633790 5011 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:51.634027 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633793 5011 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:51.634027 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633797 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:51.634027 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633801 5011 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:51.634027 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633806 5011 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:51.634027 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633810 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:51.634027 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633814 5011 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:51.634027 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633818 5011 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:51.634027 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633822 5011 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:51.634027 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633851 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:51.634027 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633856 5011 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:51.634847 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633860 5011 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:51.634847 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633863 5011 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:51.634847 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633867 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:51.634847 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633871 5011 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:51.634847 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633875 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:51.634847 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633879 5011 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:51.634847 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633888 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:51.634847 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633892 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:51.634847 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633896 5011 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:51.634847 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633900 5011 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:51.634847 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633904 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:51.634847 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633909 5011 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:51.634847 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633914 5011 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:51.634847 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633919 5011 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:51.634847 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633923 5011 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:51.634847 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633927 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:51.634847 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633932 5011 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:51.634847 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633939 5011 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:51.634847 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633946 5011 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:51.635709 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633951 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:51.635709 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633958 5011 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:51.635709 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633963 5011 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:51.635709 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633967 5011 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:51.635709 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633971 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:51.635709 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633976 5011 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:51.635709 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633982 5011 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:51.635709 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633986 5011 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:51.635709 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633990 5011 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:51.635709 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.633995 5011 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:51.635709 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634001 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:51.635709 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634005 5011 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:51.635709 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634009 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:51.635709 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634014 5011 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:51.635709 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634018 5011 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:51.635709 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634022 5011 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:51.635709 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634026 5011 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:51.635709 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634029 5011 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:51.635709 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634033 5011 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:51.635709 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634037 5011 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:51.636580 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634041 5011 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:51.636580 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634046 5011 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:51.636580 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634052 5011 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:51.636580 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634057 5011 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:51.636580 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634063 5011 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:51.636580 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634069 5011 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:51.636580 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634074 5011 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:51.636580 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634078 5011 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:51.636580 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634083 5011 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:51.636580 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634087 5011 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:51.636580 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634091 5011 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:51.636580 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634096 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:51.636580 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634100 5011 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:51.636580 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634104 5011 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:51.636580 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634108 5011 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:51.636580 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634113 5011 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:51.636580 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634117 5011 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:51.636580 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634123 5011 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:51.636580 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634127 5011 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:51.637589 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634132 5011 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:51.637589 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634137 5011 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:51.637589 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634141 5011 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:51.637589 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634145 5011 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:51.637589 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634149 5011 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:51.637589 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634154 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:51.637589 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634158 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:51.637589 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634163 5011 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:51.637589 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634167 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:51.637589 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634171 5011 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:51.637589 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634175 5011 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:51.637589 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634179 5011 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:51.637589 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634184 5011 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:51.637589 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634188 5011 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:51.637589 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634192 5011 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:51.637589 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634196 5011 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:51.637589 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634200 5011 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:51.637589 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634204 5011 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:51.637589 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634210 5011 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:51.637589 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634214 5011 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:51.638495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634220 5011 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:51.638495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634224 5011 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:51.638495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634228 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:51.638495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634233 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:51.638495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.634237 5011 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:51.638495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635492 5011 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:51.638495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635503 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:51.638495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635508 5011 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:51.638495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635513 5011 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:51.638495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635517 5011 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:51.638495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635521 5011 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:51.638495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635526 5011 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:51.638495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635530 5011 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:51.638495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635536 5011 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:51.638495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635543 5011 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:51.638495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635550 5011 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:51.638495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635555 5011 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:51.638495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635559 5011 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:51.638495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635564 5011 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:51.639437 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635570 5011 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:51.639437 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635575 5011 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:51.639437 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635580 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:51.639437 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635584 5011 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:51.639437 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635588 5011 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:51.639437 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635593 5011 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:51.639437 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635598 5011 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:51.639437 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635603 5011 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:51.639437 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635607 5011 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:51.639437 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635611 5011 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:51.639437 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635615 5011 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:51.639437 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635620 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:51.639437 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635624 5011 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:51.639437 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635628 5011 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:51.639437 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635632 5011 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:51.639437 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635636 5011 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:51.639437 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635640 5011 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:51.639437 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635645 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:51.639437 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635649 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:51.639437 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635653 5011 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:51.640270 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635658 5011 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:51.640270 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635662 5011 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:51.640270 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635666 5011 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:51.640270 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635670 5011 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:51.640270 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635674 5011 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:51.640270 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635679 5011 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:51.640270 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635683 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:51.640270 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635696 5011 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:51.640270 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635701 5011 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:51.640270 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635705 5011 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:51.640270 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635709 5011 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:51.640270 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635713 5011 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:51.640270 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635717 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:51.640270 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635721 5011 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:51.640270 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635725 5011 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:51.640270 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635728 5011 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:51.640270 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635733 5011 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:51.640270 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635737 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:51.640270 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635742 5011 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:51.640270 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635746 5011 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:51.640790 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635750 5011 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:51.640790 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635755 5011 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:51.640790 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635759 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:51.640790 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635763 5011 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:51.640790 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635767 5011 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:51.640790 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635771 5011 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:51.640790 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635775 5011 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:51.640790 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635779 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:51.640790 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635783 5011 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:51.640790 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635788 5011 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:51.640790 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635792 5011 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:51.640790 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635796 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:51.640790 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635800 5011 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:51.640790 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635804 5011 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:51.640790 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635808 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:51.640790 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635812 5011 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:51.640790 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635817 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:51.640790 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635822 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:51.640790 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635846 5011 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:51.640790 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635851 5011 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:51.641339 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635855 5011 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:51.641339 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635860 5011 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:51.641339 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635864 5011 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:51.641339 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635868 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:51.641339 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635872 5011 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:51.641339 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635876 5011 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:51.641339 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635881 5011 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:51.641339 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635885 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:51.641339 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635889 5011 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:51.641339 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635893 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:51.641339 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635901 5011 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:51.641339 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635908 5011 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:51.641339 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635913 5011 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:51.641339 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635919 5011 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:51.641339 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635923 5011 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:51.641339 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635927 5011 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:51.641339 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635931 5011 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:51.641339 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635935 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:51.641339 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635939 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:51.641339 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635943 5011 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:51.641824 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635948 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:51.641824 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635952 5011 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:51.641824 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635957 5011 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:51.641824 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635962 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:51.641824 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635966 5011 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:51.641824 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635970 5011 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:51.641824 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635974 5011 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:51.641824 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635979 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:51.641824 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635983 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:51.641824 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635987 5011 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:51.641824 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635994 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:51.641824 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.635999 5011 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:51.641824 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.636877 5011 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:51.641824 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.636888 5011 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:51.641824 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.636894 5011 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:51.641824 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.636900 5011 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:51.641824 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.636904 5011 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:51.641824 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.636909 5011 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:51.641824 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.636913 5011 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:51.642360 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.636917 5011 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:51.642360 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.636921 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:51.642360 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.636927 5011 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:51.642360 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.636931 5011 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:51.642360 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.636935 5011 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:51.642360 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.636940 5011 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:51.642360 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.636944 5011 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:51.642360 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.636948 5011 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:51.642360 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.636952 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:51.642360 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.636957 5011 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:51.642360 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.636961 5011 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:51.642360 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.636965 5011 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:51.642360 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.636969 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:51.642360 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.636974 5011 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:51.642360 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.636978 5011 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:51.642360 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.636982 5011 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:51.642360 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.636985 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:51.642360 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.636989 5011 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:51.642360 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.636993 5011 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:51.642360 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.636996 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:51.642889 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637001 5011 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:51.642889 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637005 5011 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:51.642889 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637010 5011 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:51.642889 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637015 5011 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:51.642889 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637019 5011 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:51.642889 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637023 5011 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:51.642889 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637028 5011 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:51.642889 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637032 5011 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:51.642889 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637036 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:51.642889 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637040 5011 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:51.642889 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637045 5011 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:51.642889 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637049 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:51.642889 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637053 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:51.642889 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637057 5011 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:51.642889 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637064 5011 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:51.642889 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637071 5011 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:51.642889 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637076 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:51.642889 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637081 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:51.642889 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637086 5011 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:51.642889 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637091 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:51.643382 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637096 5011 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:51.643382 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637100 5011 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:51.643382 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637104 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:51.643382 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637110 5011 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:51.643382 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637115 5011 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:51.643382 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637120 5011 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:51.643382 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637124 5011 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:51.643382 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637129 5011 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:51.643382 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637133 5011 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:51.643382 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637137 5011 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:51.643382 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637143 5011 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:51.643382 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637148 5011 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:51.643382 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637152 5011 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:51.643382 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637156 5011 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:51.643382 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637161 5011 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:51.643382 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637165 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:51.643382 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637169 5011 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:51.643382 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637173 5011 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:51.643382 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637177 5011 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:51.643854 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637182 5011 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:51.643854 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637208 5011 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:51.643854 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637215 5011 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:51.643854 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637224 5011 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:51.643854 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637229 5011 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:51.643854 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637233 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:51.643854 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637238 5011 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:51.643854 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637243 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:51.643854 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637247 5011 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:51.643854 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637251 5011 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:51.643854 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637256 5011 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:51.643854 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637260 5011 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:51.643854 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637264 5011 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:51.643854 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637268 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:51.643854 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637273 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:51.643854 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637277 5011 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:51.643854 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637281 5011 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:51.643854 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637285 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:51.643854 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637290 5011 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:51.644314 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637294 5011 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:51.644314 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637298 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:51.644314 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637302 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:51.644314 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637307 5011 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:51.644314 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637311 5011 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:51.644314 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637315 5011 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:51.644314 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637320 5011 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:51.644314 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637324 5011 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:51.644314 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637328 5011 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:51.644314 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637332 5011 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:51.644314 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637336 5011 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:51.644314 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637340 5011 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:51.644314 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637344 5011 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:51.644314 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637350 5011 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:51.644314 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637356 5011 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:51.644314 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637361 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:51.644314 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637367 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:51.644314 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637372 5011 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:51.644314 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637376 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:51.644761 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637380 5011 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:51.644761 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.637384 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:51.644761 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637527 5011 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 22:23:51.644761 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637544 5011 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 22:23:51.644761 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637554 5011 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 22:23:51.644761 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637561 5011 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 22:23:51.644761 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637568 5011 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 22:23:51.644761 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637573 5011 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 22:23:51.644761 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637580 5011 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 22:23:51.644761 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637586 5011 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 22:23:51.644761 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637591 5011 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 22:23:51.644761 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637595 5011 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 22:23:51.644761 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637603 5011 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 22:23:51.644761 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637608 5011 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 22:23:51.644761 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637612 5011 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 22:23:51.644761 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637617 5011 flags.go:64] FLAG: --cgroup-root="" Nov 01 22:23:51.644761 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637622 5011 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 22:23:51.644761 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637626 5011 flags.go:64] FLAG: --client-ca-file="" Nov 01 22:23:51.644761 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637631 5011 flags.go:64] FLAG: --cloud-provider="external" Nov 01 22:23:51.644761 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637636 5011 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 22:23:51.644761 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637641 5011 flags.go:64] FLAG: --cluster-domain="" Nov 01 22:23:51.644761 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637646 5011 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 22:23:51.644761 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637651 5011 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 22:23:51.644761 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637656 5011 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 22:23:51.645439 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637661 5011 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 22:23:51.645439 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637668 5011 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 22:23:51.645439 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637672 5011 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 22:23:51.645439 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637677 5011 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 22:23:51.645439 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637683 5011 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 22:23:51.645439 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637690 5011 flags.go:64] FLAG: --contention-profiling="false" Nov 01 22:23:51.645439 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637695 5011 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 22:23:51.645439 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637699 5011 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 22:23:51.645439 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637704 5011 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 22:23:51.645439 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637709 5011 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 22:23:51.645439 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637716 5011 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 22:23:51.645439 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637721 5011 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 22:23:51.645439 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637725 5011 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 22:23:51.645439 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637730 5011 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 22:23:51.645439 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637735 5011 flags.go:64] FLAG: --enable-server="true" Nov 01 22:23:51.645439 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637739 5011 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 22:23:51.645439 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637746 5011 flags.go:64] FLAG: --event-burst="100" Nov 01 22:23:51.645439 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637751 5011 flags.go:64] FLAG: --event-qps="50" Nov 01 22:23:51.645439 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637756 5011 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 22:23:51.645439 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637761 5011 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 22:23:51.645439 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637767 5011 flags.go:64] FLAG: --eviction-hard="" Nov 01 22:23:51.645439 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637774 5011 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 22:23:51.645439 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637779 5011 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 22:23:51.645439 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637784 5011 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 22:23:51.645439 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637788 5011 flags.go:64] FLAG: --eviction-soft="" Nov 01 22:23:51.646098 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637793 5011 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 22:23:51.646098 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637798 5011 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 22:23:51.646098 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637803 5011 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 22:23:51.646098 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637808 5011 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 22:23:51.646098 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637812 5011 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 22:23:51.646098 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637817 5011 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 22:23:51.646098 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637822 5011 flags.go:64] FLAG: --feature-gates="" Nov 01 22:23:51.646098 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637848 5011 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 22:23:51.646098 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637853 5011 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 22:23:51.646098 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637858 5011 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 22:23:51.646098 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637863 5011 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 22:23:51.646098 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637868 5011 flags.go:64] FLAG: --healthz-port="10248" Nov 01 22:23:51.646098 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637875 5011 flags.go:64] FLAG: --help="false" Nov 01 22:23:51.646098 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637880 5011 flags.go:64] FLAG: --hostname-override="ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:23:51.646098 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637885 5011 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 22:23:51.646098 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637890 5011 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 22:23:51.646098 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637895 5011 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 22:23:51.646098 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637900 5011 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 22:23:51.646098 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637905 5011 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 22:23:51.646098 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637910 5011 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 22:23:51.646098 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637915 5011 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 22:23:51.646098 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637919 5011 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 22:23:51.646098 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637924 5011 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 22:23:51.646665 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637929 5011 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 22:23:51.646665 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637934 5011 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 22:23:51.646665 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637939 5011 flags.go:64] FLAG: --kube-reserved="" Nov 01 22:23:51.646665 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637944 5011 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 22:23:51.646665 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637951 5011 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 22:23:51.646665 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637956 5011 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 22:23:51.646665 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637961 5011 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 22:23:51.646665 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637965 5011 flags.go:64] FLAG: --lock-file="" Nov 01 22:23:51.646665 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637970 5011 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 22:23:51.646665 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637975 5011 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 22:23:51.646665 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637980 5011 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 22:23:51.646665 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637989 5011 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 22:23:51.646665 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637993 5011 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 22:23:51.646665 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.637998 5011 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 22:23:51.646665 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638003 5011 flags.go:64] FLAG: --logging-format="text" Nov 01 22:23:51.646665 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638008 5011 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 22:23:51.646665 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638013 5011 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 22:23:51.646665 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638017 5011 flags.go:64] FLAG: --manifest-url="" Nov 01 22:23:51.646665 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638022 5011 flags.go:64] FLAG: --manifest-url-header="" Nov 01 22:23:51.646665 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638028 5011 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 22:23:51.646665 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638057 5011 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 22:23:51.646665 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638067 5011 flags.go:64] FLAG: --max-pods="110" Nov 01 22:23:51.646665 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638073 5011 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 22:23:51.646665 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638078 5011 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 22:23:51.646665 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638083 5011 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 22:23:51.647276 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638087 5011 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 22:23:51.647276 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638092 5011 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 22:23:51.647276 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638097 5011 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 22:23:51.647276 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638101 5011 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 22:23:51.647276 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638113 5011 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 22:23:51.647276 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638118 5011 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 22:23:51.647276 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638123 5011 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 22:23:51.647276 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638128 5011 flags.go:64] FLAG: --pod-cidr="" Nov 01 22:23:51.647276 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638132 5011 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 22:23:51.647276 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638137 5011 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 22:23:51.647276 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638142 5011 flags.go:64] FLAG: --pods-per-core="0" Nov 01 22:23:51.647276 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638148 5011 flags.go:64] FLAG: --port="10250" Nov 01 22:23:51.647276 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638154 5011 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 22:23:51.647276 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638159 5011 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0a8a9e93cb5c0d220" Nov 01 22:23:51.647276 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638164 5011 flags.go:64] FLAG: --qos-reserved="" Nov 01 22:23:51.647276 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638169 5011 flags.go:64] FLAG: --read-only-port="10255" Nov 01 22:23:51.647276 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638174 5011 flags.go:64] FLAG: --register-node="true" Nov 01 22:23:51.647276 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638179 5011 flags.go:64] FLAG: --register-with-taints="" Nov 01 22:23:51.647276 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638185 5011 flags.go:64] FLAG: --registry-burst="10" Nov 01 22:23:51.647276 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638189 5011 flags.go:64] FLAG: --registry-qps="5" Nov 01 22:23:51.647276 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638194 5011 flags.go:64] FLAG: --reserved-cpus="" Nov 01 22:23:51.647276 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638198 5011 flags.go:64] FLAG: --reserved-memory="" Nov 01 22:23:51.647276 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638204 5011 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 22:23:51.647276 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638210 5011 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 22:23:51.647276 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638214 5011 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 22:23:51.647276 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638219 5011 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 22:23:51.648004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638233 5011 flags.go:64] FLAG: --runonce="false" Nov 01 22:23:51.648004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638239 5011 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 22:23:51.648004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638248 5011 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 22:23:51.648004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638253 5011 flags.go:64] FLAG: --seccomp-default="false" Nov 01 22:23:51.648004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638260 5011 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 22:23:51.648004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638264 5011 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 22:23:51.648004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638269 5011 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 22:23:51.648004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638274 5011 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 22:23:51.648004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638279 5011 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 22:23:51.648004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638283 5011 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 22:23:51.648004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638288 5011 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 22:23:51.648004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638293 5011 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 22:23:51.648004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638298 5011 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 22:23:51.648004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638302 5011 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 22:23:51.648004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638307 5011 flags.go:64] FLAG: --system-cgroups="" Nov 01 22:23:51.648004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638312 5011 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 22:23:51.648004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638320 5011 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 22:23:51.648004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638327 5011 flags.go:64] FLAG: --tls-cert-file="" Nov 01 22:23:51.648004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638332 5011 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 22:23:51.648004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638338 5011 flags.go:64] FLAG: --tls-min-version="" Nov 01 22:23:51.648004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638344 5011 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 22:23:51.648004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638348 5011 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 22:23:51.648004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638353 5011 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 22:23:51.648004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638358 5011 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 22:23:51.648004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638363 5011 flags.go:64] FLAG: --v="2" Nov 01 22:23:51.648783 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638369 5011 flags.go:64] FLAG: --version="false" Nov 01 22:23:51.648783 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638376 5011 flags.go:64] FLAG: --vmodule="" Nov 01 22:23:51.648783 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638382 5011 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 22:23:51.648783 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.638387 5011 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 22:23:51.648783 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638604 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:51.648783 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638612 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:51.648783 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638617 5011 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:51.648783 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638621 5011 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:51.648783 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638626 5011 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:51.648783 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638630 5011 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:51.648783 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638634 5011 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:51.648783 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638638 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:51.648783 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638644 5011 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:51.648783 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638648 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:51.648783 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638652 5011 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:51.648783 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638656 5011 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:51.648783 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638661 5011 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:51.648783 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638665 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:51.648783 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638669 5011 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:51.648783 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638673 5011 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:51.649559 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638677 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:51.649559 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638681 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:51.649559 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638685 5011 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:51.649559 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638689 5011 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:51.649559 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638698 5011 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:51.649559 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638703 5011 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:51.649559 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638707 5011 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:51.649559 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638711 5011 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:51.649559 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638715 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:51.649559 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638720 5011 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:51.649559 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638724 5011 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:51.649559 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638731 5011 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:51.649559 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638738 5011 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:51.649559 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638743 5011 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:51.649559 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638748 5011 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:51.649559 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638753 5011 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:51.649559 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638757 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:51.649559 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638762 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:51.650104 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638766 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:51.650104 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638770 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:51.650104 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638774 5011 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:51.650104 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638778 5011 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:51.650104 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638782 5011 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:51.650104 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638787 5011 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:51.650104 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638791 5011 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:51.650104 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638803 5011 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:51.650104 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638807 5011 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:51.650104 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638812 5011 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:51.650104 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638816 5011 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:51.650104 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638820 5011 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:51.650104 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638824 5011 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:51.650104 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638853 5011 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:51.650104 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638857 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:51.650104 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638861 5011 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:51.650104 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638865 5011 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:51.650104 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638870 5011 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:51.650104 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638876 5011 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:51.650104 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638880 5011 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:51.650617 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638885 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:51.650617 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638889 5011 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:51.650617 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638893 5011 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:51.650617 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638897 5011 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:51.650617 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638901 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:51.650617 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638905 5011 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:51.650617 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638910 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:51.650617 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638914 5011 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:51.650617 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638918 5011 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:51.650617 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638922 5011 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:51.650617 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638926 5011 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:51.650617 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638931 5011 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:51.650617 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638935 5011 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:51.650617 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638939 5011 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:51.650617 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638943 5011 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:51.650617 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638947 5011 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:51.650617 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638951 5011 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:51.650617 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638956 5011 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:51.650617 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638960 5011 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:51.650617 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638964 5011 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:51.651204 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638969 5011 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:51.651204 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638974 5011 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:51.651204 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638978 5011 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:51.651204 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638982 5011 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:51.651204 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638987 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:51.651204 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638991 5011 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:51.651204 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.638995 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:51.651204 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639000 5011 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:51.651204 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639004 5011 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:51.651204 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639008 5011 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:51.651204 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639014 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:51.651204 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639018 5011 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:51.651204 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639022 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:51.651204 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639026 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:51.651204 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639030 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:51.651204 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639034 5011 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:51.651204 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639039 5011 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:51.651204 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639045 5011 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:51.651204 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639051 5011 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:51.651204 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639057 5011 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:51.651708 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639061 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:51.651708 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639066 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:51.651708 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639070 5011 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:51.651708 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639074 5011 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:51.651708 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639079 5011 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:51.651708 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639083 5011 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:51.651708 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639087 5011 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:51.651708 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639091 5011 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:51.651708 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639095 5011 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:51.651708 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639099 5011 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:51.651708 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639104 5011 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:51.651708 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639108 5011 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:51.651708 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.639154 5011 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 22:23:51.651708 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639210 5011 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:51.651708 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639217 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:51.651708 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639224 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:51.652249 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639229 5011 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:51.652249 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639233 5011 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:51.652249 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639238 5011 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:51.652249 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639243 5011 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:51.652249 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639247 5011 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:51.652249 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639251 5011 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:51.652249 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639257 5011 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:51.652249 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639261 5011 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:51.652249 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639265 5011 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:51.652249 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639270 5011 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:51.652249 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639274 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:51.652249 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639279 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:51.652249 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639283 5011 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:51.652249 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639287 5011 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:51.652249 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639291 5011 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:51.652249 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639295 5011 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:51.652249 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639299 5011 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:51.652249 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639304 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:51.652249 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639309 5011 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:51.652249 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639313 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:51.652878 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639317 5011 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:51.652878 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639321 5011 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:51.652878 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639325 5011 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:51.652878 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639329 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:51.652878 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639333 5011 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:51.652878 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639337 5011 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:51.652878 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639342 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:51.652878 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639346 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:51.652878 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639350 5011 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:51.652878 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639356 5011 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:51.652878 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639361 5011 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:51.652878 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639374 5011 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:51.652878 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639380 5011 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:51.652878 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639384 5011 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:51.652878 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639388 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:51.652878 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639392 5011 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:51.652878 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639396 5011 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:51.652878 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639404 5011 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:51.652878 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639414 5011 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:51.653455 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639420 5011 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:51.653455 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639425 5011 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:51.653455 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639431 5011 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:51.653455 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639435 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:51.653455 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639439 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:51.653455 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639443 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:51.653455 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639447 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:51.653455 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639451 5011 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:51.653455 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639455 5011 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:51.653455 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639459 5011 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:51.653455 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639463 5011 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:51.653455 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639467 5011 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:51.653455 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639472 5011 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:51.653455 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639476 5011 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:51.653455 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639480 5011 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:51.653455 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639485 5011 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:51.653455 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639489 5011 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:51.653455 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639493 5011 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:51.653455 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639497 5011 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:51.653963 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639501 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:51.653963 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639505 5011 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:51.653963 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639510 5011 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:51.653963 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639516 5011 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:51.653963 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639521 5011 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:51.653963 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639533 5011 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:51.653963 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639538 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:51.653963 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639542 5011 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:51.653963 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639546 5011 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:51.653963 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639550 5011 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:51.653963 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639554 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:51.653963 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639558 5011 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:51.653963 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639565 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:51.653963 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639569 5011 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:51.653963 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639573 5011 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:51.653963 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639577 5011 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:51.653963 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639582 5011 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:51.653963 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639586 5011 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:51.653963 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639590 5011 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:51.653963 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639594 5011 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:51.654451 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639598 5011 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:51.654451 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639602 5011 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:51.654451 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639606 5011 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:51.654451 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639611 5011 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:51.654451 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639615 5011 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:51.654451 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639619 5011 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:51.654451 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639623 5011 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:51.654451 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639627 5011 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:51.654451 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639631 5011 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:51.654451 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639636 5011 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:51.654451 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639641 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:51.654451 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639645 5011 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:51.654451 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639649 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:51.654451 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639654 5011 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:51.654451 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639658 5011 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:51.654451 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639663 5011 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:51.654451 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639668 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:51.654451 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639672 5011 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:51.654451 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639677 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:51.654451 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639681 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:51.655005 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639685 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:51.655005 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639689 5011 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:51.655005 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639694 5011 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:51.655005 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639701 5011 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:51.655005 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.639708 5011 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:51.655005 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.643656 5011 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 22:23:51.655005 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.643673 5011 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 22:23:51.655005 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.643691 5011 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 22:23:51.655005 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.643696 5011 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 22:23:51.655005 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643765 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:51.655005 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643770 5011 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:51.655005 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643773 5011 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:51.655005 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643776 5011 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:51.655005 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643779 5011 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:51.655005 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643782 5011 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:51.655005 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643784 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:51.655005 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643787 5011 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:51.655005 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643790 5011 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:51.655005 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643793 5011 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:51.655005 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643795 5011 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:51.655495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643798 5011 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:51.655495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643800 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:51.655495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643803 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:51.655495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643806 5011 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:51.655495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643808 5011 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:51.655495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643811 5011 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:51.655495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643814 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:51.655495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643816 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:51.655495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643819 5011 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:51.655495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643821 5011 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:51.655495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643823 5011 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:51.655495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643852 5011 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:51.655495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643856 5011 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:51.655495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643859 5011 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:51.655495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643862 5011 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:51.655495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643864 5011 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:51.655495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643867 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:51.655495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643870 5011 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:51.655495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643874 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:51.655495 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643877 5011 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:51.656088 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643880 5011 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:51.656088 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643882 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:51.656088 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643885 5011 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:51.656088 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643888 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:51.656088 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643890 5011 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:51.656088 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643893 5011 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:51.656088 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643896 5011 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:51.656088 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643898 5011 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:51.656088 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643900 5011 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:51.656088 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643903 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:51.656088 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643905 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:51.656088 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643908 5011 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:51.656088 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643910 5011 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:51.656088 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643913 5011 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:51.656088 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643916 5011 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:51.656088 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643918 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:51.656088 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643921 5011 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:51.656088 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643923 5011 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:51.656088 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643927 5011 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:51.656542 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643931 5011 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:51.656542 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643934 5011 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:51.656542 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643937 5011 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:51.656542 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643940 5011 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:51.656542 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643943 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:51.656542 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643945 5011 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:51.656542 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643948 5011 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:51.656542 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643950 5011 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:51.656542 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643953 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:51.656542 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643955 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:51.656542 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643958 5011 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:51.656542 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643960 5011 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:51.656542 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643963 5011 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:51.656542 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643966 5011 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:51.656542 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643969 5011 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:51.656542 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643971 5011 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:51.656542 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643974 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:51.656542 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643976 5011 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:51.656542 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643978 5011 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:51.657015 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643981 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:51.657015 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643983 5011 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:51.657015 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643986 5011 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:51.657015 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643988 5011 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:51.657015 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643991 5011 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:51.657015 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643993 5011 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:51.657015 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643996 5011 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:51.657015 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.643998 5011 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:51.657015 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644002 5011 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:51.657015 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644006 5011 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:51.657015 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644010 5011 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:51.657015 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644013 5011 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:51.657015 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644016 5011 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:51.657015 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644018 5011 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:51.657015 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644021 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:51.657015 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644023 5011 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:51.657015 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644026 5011 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:51.657015 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644028 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:51.657015 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644031 5011 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:51.657464 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644033 5011 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:51.657464 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644036 5011 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:51.657464 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644038 5011 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:51.657464 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644041 5011 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:51.657464 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644044 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:51.657464 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644046 5011 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:51.657464 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644049 5011 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:51.657464 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644052 5011 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:51.657464 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644054 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:51.657464 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644057 5011 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:51.657464 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644059 5011 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:51.657464 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644062 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:51.657464 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644064 5011 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:51.657464 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644066 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:51.657464 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644069 5011 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:51.657464 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644072 5011 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:51.657464 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644074 5011 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:51.657464 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644076 5011 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:51.657931 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.644104 5011 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 22:23:51.657931 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644268 5011 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:51.657931 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644274 5011 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:51.657931 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644278 5011 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:51.657931 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644282 5011 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:51.657931 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644285 5011 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:51.657931 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644288 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:51.657931 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644290 5011 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:51.657931 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644293 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:51.657931 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644295 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:51.657931 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644298 5011 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:51.657931 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644300 5011 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:51.657931 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644303 5011 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:51.657931 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644305 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:51.657931 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644308 5011 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:51.658282 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644310 5011 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:51.658282 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644313 5011 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:51.658282 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644316 5011 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:51.658282 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644319 5011 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:51.658282 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644321 5011 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:51.658282 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644324 5011 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:51.658282 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644327 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:51.658282 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644330 5011 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:51.658282 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644333 5011 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:51.658282 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644335 5011 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:51.658282 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644338 5011 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:51.658282 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644340 5011 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:51.658282 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644342 5011 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:51.658282 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644345 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:51.658282 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644347 5011 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:51.658282 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644350 5011 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:51.658282 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644352 5011 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:51.658282 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644355 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:51.658282 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644357 5011 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:51.658754 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644360 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:51.658754 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644362 5011 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:51.658754 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644364 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:51.658754 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644367 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:51.658754 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644369 5011 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:51.658754 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644372 5011 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:51.658754 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644374 5011 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:51.658754 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644376 5011 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:51.658754 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644379 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:51.658754 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644381 5011 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:51.658754 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644384 5011 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:51.658754 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644386 5011 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:51.658754 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644389 5011 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:51.658754 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644391 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:51.658754 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644393 5011 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:51.658754 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644396 5011 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:51.658754 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644398 5011 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:51.658754 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644401 5011 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:51.658754 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644403 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:51.658754 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644405 5011 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:51.659247 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644408 5011 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:51.659247 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644411 5011 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:51.659247 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644413 5011 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:51.659247 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644416 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:51.659247 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644418 5011 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:51.659247 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644421 5011 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:51.659247 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644423 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:51.659247 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644426 5011 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:51.659247 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644428 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:51.659247 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644430 5011 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:51.659247 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644433 5011 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:51.659247 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644436 5011 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:51.659247 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644438 5011 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:51.659247 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644440 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:51.659247 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644444 5011 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:51.659247 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644448 5011 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:51.659247 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644451 5011 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:51.659247 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644453 5011 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:51.659247 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644456 5011 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:51.659247 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644458 5011 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:51.659716 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644461 5011 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:51.659716 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644463 5011 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:51.659716 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644466 5011 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:51.659716 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644468 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:51.659716 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644471 5011 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:51.659716 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644473 5011 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:51.659716 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644475 5011 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:51.659716 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644478 5011 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:51.659716 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644480 5011 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:51.659716 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644482 5011 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:51.659716 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644485 5011 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:51.659716 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644487 5011 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:51.659716 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644489 5011 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:51.659716 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644492 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:51.659716 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644495 5011 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:51.659716 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644497 5011 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:51.659716 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644499 5011 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:51.659716 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644502 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:51.659716 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644504 5011 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:51.659716 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644506 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:51.660259 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644509 5011 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:51.660259 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644511 5011 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:51.660259 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644516 5011 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:51.660259 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644519 5011 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:51.660259 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644521 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:51.660259 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644523 5011 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:51.660259 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644526 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:51.660259 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644528 5011 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:51.660259 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644531 5011 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:51.660259 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644533 5011 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:51.660259 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644535 5011 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:51.660259 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644538 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:51.660259 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644540 5011 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:51.660259 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.644563 5011 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 22:23:51.660259 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644595 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:51.660259 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644598 5011 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:51.660650 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644601 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:51.660650 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644604 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:51.660650 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644606 5011 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:51.660650 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644609 5011 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:51.660650 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644611 5011 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:51.660650 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644614 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:51.660650 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644616 5011 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:51.660650 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644619 5011 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:51.660650 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644621 5011 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:51.660650 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644624 5011 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:51.660650 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644626 5011 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:51.660650 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644629 5011 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:51.660650 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644632 5011 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:51.660650 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644634 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:51.660650 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644637 5011 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:51.660650 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644639 5011 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:51.660650 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644642 5011 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:51.660650 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644644 5011 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:51.660650 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644647 5011 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:51.661112 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644651 5011 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:51.661112 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644654 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:51.661112 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644657 5011 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:51.661112 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644660 5011 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:51.661112 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644662 5011 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:51.661112 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644664 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:51.661112 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644667 5011 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:51.661112 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644669 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:51.661112 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644672 5011 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:51.661112 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644674 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:51.661112 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644677 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:51.661112 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644679 5011 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:51.661112 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644682 5011 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:51.661112 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644684 5011 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:51.661112 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644687 5011 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:51.661112 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644689 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:51.661112 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644692 5011 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:51.661112 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644694 5011 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:51.661112 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644697 5011 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:51.661552 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644699 5011 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:51.661552 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644701 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:51.661552 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644704 5011 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:51.661552 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644706 5011 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:51.661552 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644709 5011 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:51.661552 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644711 5011 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:51.661552 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644714 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:51.661552 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644717 5011 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:51.661552 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644719 5011 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:51.661552 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644721 5011 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:51.661552 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644724 5011 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:51.661552 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644726 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:51.661552 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644729 5011 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:51.661552 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644732 5011 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:51.661552 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644735 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:51.661552 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644737 5011 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:51.661552 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644740 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:51.661552 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644742 5011 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:51.661552 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644745 5011 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:51.661552 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644747 5011 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:51.662053 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644750 5011 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:51.662053 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644752 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:51.662053 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644756 5011 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:51.662053 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644761 5011 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:51.662053 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644763 5011 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:51.662053 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644766 5011 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:51.662053 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644769 5011 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:51.662053 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644771 5011 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:51.662053 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644774 5011 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:51.662053 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644777 5011 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:51.662053 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644780 5011 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:51.662053 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644782 5011 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:51.662053 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644784 5011 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:51.662053 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644787 5011 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:51.662053 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644789 5011 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:51.662053 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644792 5011 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:51.662053 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644794 5011 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:51.662053 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644796 5011 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:51.662053 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644799 5011 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:51.662053 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644801 5011 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:51.662543 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644804 5011 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:51.662543 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644806 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:51.662543 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644809 5011 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:51.662543 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644811 5011 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:51.662543 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644814 5011 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:51.662543 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644818 5011 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:51.662543 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644821 5011 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:51.662543 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644823 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:51.662543 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644825 5011 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:51.662543 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644844 5011 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:51.662543 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644847 5011 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:51.662543 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644850 5011 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:51.662543 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644852 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:51.662543 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644855 5011 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:51.662543 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644857 5011 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:51.662543 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644860 5011 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:51.662543 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644863 5011 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:51.662543 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644865 5011 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:51.662543 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644868 5011 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:51.662543 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644870 5011 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:51.663188 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644872 5011 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:51.663188 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644875 5011 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:51.663188 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644879 5011 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:51.663188 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644882 5011 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:51.663188 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644885 5011 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:51.663188 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:51.644890 5011 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:51.663188 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.644958 5011 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 22:23:51.663188 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.646001 5011 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 01 22:23:51.663188 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.646051 5011 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 01 22:23:51.663188 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.646233 5011 server.go:1014] "Starting client certificate rotation" Nov 01 22:23:51.663188 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.646364 5011 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 22:23:51.663188 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.646404 5011 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 22:17:40 +0000 UTC" deadline="2028-07-19 05:59:22.772713862 +0000 UTC" Nov 01 22:23:51.663188 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.646444 5011 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15007h35m31.126273313s" Nov 01 22:23:51.663188 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.646667 5011 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 22:23:51.663188 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.647763 5011 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 22:23:51.663188 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.650443 5011 log.go:25] "Validated CRI v1 runtime API" Nov 01 22:23:51.663579 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.652020 5011 log.go:25] "Validated CRI v1 image API" Nov 01 22:23:51.663579 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.652676 5011 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 22:23:51.663579 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.654383 5011 fs.go:135] Filesystem UUIDs: map[259c1a59-854d-48cd-a40f-803aed05e1f1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e232985b-20c5-444c-8077-afd447b52533:/dev/nvme0n1p3] Nov 01 22:23:51.663689 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.654404 5011 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01c5e91dacb94194c3d671b9a66d0421dc6640b674a9d907d1ebfb40ce4e8632/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01c5e91dacb94194c3d671b9a66d0421dc6640b674a9d907d1ebfb40ce4e8632/userdata/shm major:0 minor:310 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0dc901321c676181e22a19ae8becbeb76d04ef9b84ee2856e6a59c9ffffbbae2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0dc901321c676181e22a19ae8becbeb76d04ef9b84ee2856e6a59c9ffffbbae2/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1d8b4c5a4a30cc4854e7d670f40126715432e1fba59c5f74cf836857393e1784/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1d8b4c5a4a30cc4854e7d670f40126715432e1fba59c5f74cf836857393e1784/userdata/shm major:0 minor:57 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/25bfba8acd1e8775dfcc86ff3f4e202898c47daba1a52956e44466ce2d833b47/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/25bfba8acd1e8775dfcc86ff3f4e202898c47daba1a52956e44466ce2d833b47/userdata/shm major:0 minor:255 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2f3d84313e3ca3144e9101369e8768171707b39c1a5069cface47f369f261a6e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2f3d84313e3ca3144e9101369e8768171707b39c1a5069cface47f369f261a6e/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46c075f064de3dcfb4faafb0bd55cdc02e69e0867346a8cd3c44a165829abd38/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46c075f064de3dcfb4faafb0bd55cdc02e69e0867346a8cd3c44a165829abd38/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5cf72dade1beccabe52d7cdfe6ddac10ac191dbcd514ce18ffb70d597769e61c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5cf72dade1beccabe52d7cdfe6ddac10ac191dbcd514ce18ffb70d597769e61c/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/655cd6a658fa393611ebf9a14fa3a90a105d9e3e7f27469a5ee7addd752db01d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/655cd6a658fa393611ebf9a14fa3a90a105d9e3e7f27469a5ee7addd752db01d/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7385d8d2150889036d38eaa7fc4e655bf36d9fb0ee541e0edca952a91e2a79c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7385d8d2150889036d38eaa7fc4e655bf36d9fb0ee541e0edca952a91e2a79c7/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/95acb05c8e561a0f8d3fe7df6dddd4812cb7042898b98ff64a446d0f4329b4c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/95acb05c8e561a0f8d3fe7df6dddd4812cb7042898b98ff64a446d0f4329b4c3/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9bf142e2f20179006f748e6798d0c2f95d670472af66e9537351839e4c1aaa45/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9bf142e2f20179006f748e6798d0c2f95d670472af66e9537351839e4c1aaa45/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e5f098e5bd56bb34954af334e7fa0786eb3399bd7c3362f0506d074be9a8579/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e5f098e5bd56bb34954af334e7fa0786eb3399bd7c3362f0506d074be9a8579/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/abacbe9c358eb7c91c09dbba58a8c965c778bac8677000afff60f2a0ecc3275b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/abacbe9c358eb7c91c09dbba58a8c965c778bac8677000afff60f2a0ecc3275b/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/af19768f02c54b9ae643154322ededc99c4bbbb36ffce3831f3930c16bc8fdc0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/af19768f02c54b9ae643154322ededc99c4bbbb36ffce3831f3930c16bc8fdc0/userdata/shm major:0 minor:257 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c048598f70f7855d3d2d4e4bc8cae60c611b1a3384a3de61ef4b6064fe9f6742/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c048598f70f7855d3d2d4e4bc8cae60c611b1a3384a3de61ef4b6064fe9f6742/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ca29d6a9e0c4f4af79bc2de4727b64eabf75000f0040f0564f732c7ee247157a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ca29d6a9e0c4f4af79bc2de4727b64eabf75000f0040f0564f732c7ee247157a/userdata/shm major:0 minor:253 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/caa1cde0c045e60f020bda4fa2b9c477955b5b0f1ed5dda0dc33c7b8df8916b6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/caa1cde0c045e60f020bda4fa2b9c477955b5b0f1ed5dda0dc33c7b8df8916b6/userdata/shm major:0 minor:58 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e23e2763a3591380a493e03e4209f502c432d0e813a2fcfb32604c549226ec42/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e23e2763a3591380a493e03e4209f502c432d0e813a2fcfb32604c549226ec42/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f0222f39630c858fec04d8f23a314c7c5687f9e0d827064de1b96d54d4bccdeb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f0222f39630c858fec04d8f23a314c7c5687f9e0d827064de1b96d54d4bccdeb/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f893907a88ffd76773a15165a7030c7870cbd584687312cc74d847dc10358a55/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f893907a88ffd76773a15165a7030c7870cbd584687312cc74d847dc10358a55/userdata/shm major:0 minor:249 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14cdd491-7383-4cc5-ac4f-60a75110d764/volumes/kubernetes.io~projected/kube-api-access-n9q9h:{mountpoint:/var/lib/kubelet/pods/14cdd491-7383-4cc5-ac4f-60a75110d764/volumes/kubernetes.io~projected/kube-api-access-n9q9h major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/179a164c-f6dd-42f9-9428-f241f81afff5/volumes/kubernetes.io~projected/kube-api-access-sstcj:{mountpoint:/var/lib/kubelet/pods/179a164c-f6dd-42f9-9428-f241f81afff5/volumes/kubernetes.io~projected/kube-api-access-sstcj major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3880904a-615b-44ac-9a2d-7b0f7ad31e0c/volumes/kubernetes.io~projected/kube-api-access-8ndlv:{mountpoint:/var/lib/kubelet/pods/3880904a-615b-44ac-9a2d-7b0f7ad31e0c/volumes/kubernetes.io~projected/kube-api-access-8ndlv major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3af15ebe-c67a-40c5-bac0-24410542413b/volumes/kubernetes.io~projected/kube-api-access-nkg5t:{mountpoint:/var/lib/kubelet/pods/3af15ebe-c67a-40c5-bac0-24410542413b/volumes/kubernetes.io~projected/kube-api-access-nkg5t major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bed4cac-d251-49be-9eb6-ce5cdb6852de/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/3bed4cac-d251-49be-9eb6-ce5cdb6852de/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3eeb8650-24f6-4e09-95be-794fe6a925bc/volumes/kubernetes.io~projected/kube-api-access-tdvfj:{mountpoint:/var/lib/kubelet/pods/3eeb8650-24f6-4e09-95be-794fe6a925bc/volumes/kubernetes.io~projected/kube-api-access-tdvfj major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3eeb8650-24f6-4e09-95be-794fe6a925bc/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/3eeb8650-24f6-4e09-95be-794fe6a925bc/volumes/kubernetes.io~secret/signing-key major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8/volumes/kubernetes.io~projected/kube-api-access-mqg4h:{mountpoint:/var/lib/kubelet/pods/3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8/volumes/kubernetes.io~projected/kube-api-access-mqg4h major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45d5cfa6-bf31-4fad-8003-d28fe9a37183/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/45d5cfa6-bf31-4fad-8003-d28fe9a37183/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45d5cfa6-bf31-4fad-8003-d28fe9a37183/volumes/kubernetes.io~projected/kube-api-access-kw4f7:{mountpoint:/var/lib/kubelet/pods/45d5cfa6-bf31-4fad-8003-d28fe9a37183/volumes/kubernetes.io~projected/kube-api-access-kw4f7 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45d5cfa6-bf31-4fad-8003-d28fe9a37183/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/45d5cfa6-bf31-4fad-8003-d28fe9a37183/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54ac1aee-48aa-4f54-ae3c-9d5d4e377e12/volumes/kubernetes.io~projected/kube-api-access-64mk6:{mountpoint:/var/lib/kubelet/pods/54ac1aee-48aa-4f54-ae3c-9d5d4e377e12/volumes/kubernetes.io~projected/kube-api-access-64mk6 major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c959caa-7239-4b81-8c19-c867fc6c48e6/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/5c959caa-7239-4b81-8c19-c867fc6c48e6/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c959caa-7239-4b81-8c19-c867fc6c48e6/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/5c959caa-7239-4b81-8c19-c867fc6c48e6/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c959caa-7239-4b81-8c19-c867fc6c48e6/volumes/kubernetes.io~projected/kube-api-access-s67jv:{mountpoint:/var/lib/kubelet/pods/5c959caa-7239-4b81-8c19-c867fc6c48e6/volumes/kubernetes.io~projected/kube-api-access-s67jv major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/682b0c5d-fa6f-4a17-8b72-a1f224cf5fec/volumes/kubernetes.io~projected/kube-api-access-fd2dd:{mountpoint:/var/lib/kubelet/pods/682b0c5d-fa6f-4a17-8b72-a1f224cf5fec/volumes/kubernetes.io~projected/kube-api-access-fd2dd major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9af8b8a8-47f6-4576-87db-2f0253903e5b/volumes/kubernetes.io~projected/kube-api-access-h8wg9:{mountpoint:/var/lib/kubelet/pods/9af8b8a8-47f6-4576-87db-2f0253903e5b/volumes/kubernetes.io~projected/kube-api-access-h8wg9 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9af8b8a8-47f6-4576-87db-2f0253903e5b/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/9af8b8a8-47f6-4576-87db-2f0253903e5b/volumes/kubernetes.io~secret/metrics-tls major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3a3cc56-6cfb-4092-ba39-5c237ad58c77/volumes/kubernetes.io~projected/kube-api-access-qjzrz:{mountpoint:/var/lib/kubelet/pods/b3a3cc56-6cfb-4092-ba39-5c237ad58c77/volumes/kubernetes.io~projected/kube-api-access-qjzrz major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3a3cc56-6cfb-4092-ba39-5c237ad58c77/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/b3a3cc56-6cfb-4092-ba39-5c237ad58c77/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b439b405-8bca-4ce8-980d-a076cb2f11bf/volumes/kubernetes.io~projected/kube-api-access-lxhl8:{mountpoint:/var/lib/kubelet/pods/b439b405-8bca-4ce8-980d-a076cb2f11bf/volumes/kubernetes.io~projected/kube-api-access-lxhl8 major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc3de28d-1139-4677-81db-a114c8867933/volumes/kubernetes.io~projected/kube-api-access-6nvbs:{mountpoint:/var/lib/kubelet/pods/cc3de28d-1139-4677-81db-a114c8867933/volumes/kubernetes.io~projected/kube-api-access-6nvbs major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc3de28d-1139-4677-81db-a114c8867933/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/cc3de28d-1139-4677-81db-a114c8867933/volumes/kubernetes.io~secret/metrics-certs major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8/volumes/kubernetes.io~projected/kube-api-access-gwr44:{mountpoint:/var/lib/kubelet/pods/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8/volumes/kubernetes.io~projected/kube-api-access-gwr44 major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb7b37b5-9639-45da-b90e-e08dcfac93f3/volumes/kubernetes.io~projected/kube-api-access-vm2vf:{mountpoint:/var/lib/kubelet/pods/eb7b37b5-9639-45da-b90e-e08dcfac93f3/volumes/kubernetes.io~projected/kube-api-access-vm2vf major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb7b37b5-9639-45da-b90e-e08dcfac93f3/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/eb7b37b5-9639-45da-b90e-e08dcfac93f3/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f71460db-658f-45ed-980f-da65509e2bc5/volumes/kubernetes.io~projected/kube-api-access-pzs82:{mountpoint:/var/lib/kubelet/pods/f71460db-658f-45ed-980f-da65509e2bc5/volumes/kubernetes.io~projected/kube-api-access-pzs82 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa603801-6c30-4cf1-ace8-30c8723e4a1e/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/fa603801-6c30-4cf1-ace8-30c8723e4a1e/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:248 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/1ead1de019f5e9a6ce09e65fe8beee4d3238a329df3db5d7502eb2445b87030f/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/8d08e81947f4476ca7ca5ab1b3f447b26ee6429f07b941db1744371054385c97/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/c86339f688b29d2dfb91ff10bd7f5b27b24906a77e19fbcc79d7087e59c2e05c/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/5376726766a6821c59ee7a7435efbc99c2b7cc810af2e134b450886a956a58b7/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/3572dd81a7663e59b92958a9092683505ef35f147b74bd16a52ce4efa7dfe3c3/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-116:{mountpoint:/var/lib/containers/storage/overlay/f3657d655a13bd22316874d9a2105b1a61e7b400a368b9ff6972b357ea197d2a/merged major:0 minor:116 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/c2706b5668ba3c90ff9e07aa1435b35e198c40e6ed47de3ca82ea276fdcec28c/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/11dc59acf018dbacb11870569dad9782447a6af6ad605b228e7f7aabda6c7255/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/f08802d321de69de9a04dc9dc7754c951c0b91949bc11df889df644b3fa2dc90/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/7f88546637bef042b4b80ebbed806a9b137e2ee77843e79913a7c954f3996b72/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/1b0c8cb718ea74b9dcf1b95b78dc4a0ea1ba573ad7c5dcaf5e8b1619d41ad8cf/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/2dc1e6dfffa66984c2b0cd98a579b19db28b16714f5131d14f87b7538ada4033/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/c919e3304979704417e7a7f5b3139f43c2f9b12b5c65ea657b465df4a38136d3/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/7de037a2789604ced9862a72a8877a8224860ca7ac0a0578c5cc7dbbcb50e920/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/88f663d51252720b9cd6cae549ba6548fb08694fa5473d40d4410d6d13f6b4aa/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/613daa7d481a6c9d40736da1e90491dd4d9a13238e7bf9eb0a4a1ba9c9d93463/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/aa9b849e87c9ca1cb36ad9596c18e6a282bddaf76fb3dd3b2bc560bd34d3af66/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/fa4d43407f8d127ae7366600cb86b964470d3d53761eb0136ceaf67b007fad7e/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/7785631b60c6a313570626e4c1ba7cae029e41353f8c9681fa1fadc1471af77a/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/4d01bb194895c2d7a238e7fa8ce19aba0b75d7eaf6143cfac47e9c4733da63b2/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/e7b148898fb5c385f228d8045f3dd47c8d9ae2f252ed39e84616fa1932d03856/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/988d87547885ee024d64021f023f86ee58a79ef42f7fa4f65918a61240affaef/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/4af222ea271a906660ecfb1883ad5c24a8eb5c5d7badd3bf8dd17c3359c828ed/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/d0892de0f5d769e02fc6c5da039940712521a3a94610504fae6f77f709cd4993/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/a38a752247438b08848ef88b3bbba83fe1d98fdbe68791ae4f21a00bd97ed744/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/6eb98394565fde17c2a71e22e1602166eb4eb3c507f7d7e0caf08f530322cb08/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/933e37aa2faa8f1ee320cb1846f469582ebe2cf7e044988c7bc0473b96a6b553/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/bced22e8291d9bf47caadefa822306ed7def455890a39a8476c872e8878c28ca/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/780d8d1d54a96cdb696f404b3bd451e5fc9e616fd6119f484f7eddd7c5499d47/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/ab124030a90261aed2a90a96d5ae45cd2ec1374b68ba97e896a29f344808480e/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/5e26ddf7db4817e2acf5ae568daea4c6d645b440376dff1ddc9af08359f7869b/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/d2abe16642618c7a490024bb7f6ef28dc9d6fc3c5e96b39471ebc7d50fafaacc/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/1bf1373ead678e97dd170f2e956d93ff9846a15682c6f33c28b75e525b75e092/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/98b34287a7ad286c338885f25480b6ddcb140b90597405bd0d8ea4c4d0b22851/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/fd022c9bba375d64b696ed6fa4b0abbbfe5b191d3b393a54dccce44bcfea5790/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/07d73b4db82679394043e4c6020b6b01e9d842e5e05eb553f6192b4d7ea5502a/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/72d78ebce5c5ac77e58c372eb00ab0e84d4ec48461931f8d16729f35d9d4e535/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/704025d46e2ca9ab47ce9b3d16f9a2542e78da2749f86571f0d4e8763785243a/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/e48e7a0290792eccae65207e53f9969e1a5884f40466a339f44772e63d213c93/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/87fee0799b1b8b40d404dfecf9a0fbb62361b14612c830bf21c757fb3fb75089/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/f34532573126e7532f41ebae42204d881ce7866b9a91798a2ea8cc12e0835d1d/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/7f9e46754ca9b4c023ac8c8af625d229e8274457b340320d5ed11024da6bedc8/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/dd8a483da97802d00fe68a10e75a969ebd2679f5e9cbba3274c8cd55dd288e1f/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-62:{mountpoint:/var/lib/containers/storage/overlay/b62b1cb1e113682e5abc77ed94d68b211d990c12d32d227e0c2ebad2bb4564d6/merged major:0 minor:62 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/86b9a19ac1003f135a41da63df209d074e05941272a2358bcd131653a46194e6/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/e6078457d32c92f904ad25223873571b18439cf32f1d04747c887ceb95f2e47d/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/912cc4886c5cc88110578681ca8f25dce4f4bd6fba5d047959448bc0bd73311e/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/8ea1346ed90b7365d13c12a45debc60db9b443d1833c277699613dc70f771b97/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/9a69dafc2945e0f49b39b04a2bc766f17b0a4f0bd50dd3b118a895425dd44848/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/035cf5705009df86679fe47d1b155727cf91f60da5862bcc7e6ece4303cbf12a/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/5cbdac8319b3ecf3408144dd769cbd643d1365123a0785b82c1128d877ceb382/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/7705a53c555ca1d11be28daaf8e1a8de2f4bd8512752941f9133b8128260bd81/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/324fd8c226a97c975eb753d3a39cf9780ab4aaf98d6730946075012a1145bb55/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 01 22:23:51.663887 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.661170 5011 manager.go:217] Machine: {Timestamp:2026-11-01 22:23:51.660709054 +0000 UTC m=+0.065299676 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100231 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec201604da364b01a05e96d1f942217b SystemUUID:ec201604-da36-4b01-a05e-96d1f942217b BootID:0b572456-2c6f-4dac-b385-51e3e47396b1 Filesystems:[{Device:/var/lib/kubelet/pods/45d5cfa6-bf31-4fad-8003-d28fe9a37183/volumes/kubernetes.io~projected/kube-api-access-kw4f7 DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-62 DeviceMajor:0 DeviceMinor:62 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9bf142e2f20179006f748e6798d0c2f95d670472af66e9537351839e4c1aaa45/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/14cdd491-7383-4cc5-ac4f-60a75110d764/volumes/kubernetes.io~projected/kube-api-access-n9q9h DeviceMajor:0 DeviceMinor:52 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/682b0c5d-fa6f-4a17-8b72-a1f224cf5fec/volumes/kubernetes.io~projected/kube-api-access-fd2dd DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/45d5cfa6-bf31-4fad-8003-d28fe9a37183/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/b3a3cc56-6cfb-4092-ba39-5c237ad58c77/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e5f098e5bd56bb34954af334e7fa0786eb3399bd7c3362f0506d074be9a8579/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-116 DeviceMajor:0 DeviceMinor:116 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb7b37b5-9639-45da-b90e-e08dcfac93f3/volumes/kubernetes.io~projected/kube-api-access-vm2vf DeviceMajor:0 DeviceMinor:206 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fa603801-6c30-4cf1-ace8-30c8723e4a1e/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:248 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b3a3cc56-6cfb-4092-ba39-5c237ad58c77/volumes/kubernetes.io~projected/kube-api-access-qjzrz DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5cf72dade1beccabe52d7cdfe6ddac10ac191dbcd514ce18ffb70d597769e61c/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cc3de28d-1139-4677-81db-a114c8867933/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:251 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ca29d6a9e0c4f4af79bc2de4727b64eabf75000f0040f0564f732c7ee247157a/userdata/shm DeviceMajor:0 DeviceMinor:253 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01c5e91dacb94194c3d671b9a66d0421dc6640b674a9d907d1ebfb40ce4e8632/userdata/shm DeviceMajor:0 DeviceMinor:310 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8/volumes/kubernetes.io~projected/kube-api-access-gwr44 DeviceMajor:0 DeviceMinor:302 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5c959caa-7239-4b81-8c19-c867fc6c48e6/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/95acb05c8e561a0f8d3fe7df6dddd4812cb7042898b98ff64a446d0f4329b4c3/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/abacbe9c358eb7c91c09dbba58a8c965c778bac8677000afff60f2a0ecc3275b/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8/volumes/kubernetes.io~projected/kube-api-access-mqg4h DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/45d5cfa6-bf31-4fad-8003-d28fe9a37183/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3880904a-615b-44ac-9a2d-7b0f7ad31e0c/volumes/kubernetes.io~projected/kube-api-access-8ndlv DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46c075f064de3dcfb4faafb0bd55cdc02e69e0867346a8cd3c44a165829abd38/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb7b37b5-9639-45da-b90e-e08dcfac93f3/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:280 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0dc901321c676181e22a19ae8becbeb76d04ef9b84ee2856e6a59c9ffffbbae2/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2f3d84313e3ca3144e9101369e8768171707b39c1a5069cface47f369f261a6e/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b439b405-8bca-4ce8-980d-a076cb2f11bf/volumes/kubernetes.io~projected/kube-api-access-lxhl8 DeviceMajor:0 DeviceMinor:252 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:301 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/179a164c-f6dd-42f9-9428-f241f81afff5/volumes/kubernetes.io~projected/kube-api-access-sstcj DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/655cd6a658fa393611ebf9a14fa3a90a105d9e3e7f27469a5ee7addd752db01d/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7385d8d2150889036d38eaa7fc4e655bf36d9fb0ee541e0edca952a91e2a79c7/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f893907a88ffd76773a15165a7030c7870cbd584687312cc74d847dc10358a55/userdata/shm DeviceMajor:0 DeviceMinor:249 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1d8b4c5a4a30cc4854e7d670f40126715432e1fba59c5f74cf836857393e1784/userdata/shm DeviceMajor:0 DeviceMinor:57 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cc3de28d-1139-4677-81db-a114c8867933/volumes/kubernetes.io~projected/kube-api-access-6nvbs DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9af8b8a8-47f6-4576-87db-2f0253903e5b/volumes/kubernetes.io~projected/kube-api-access-h8wg9 DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3eeb8650-24f6-4e09-95be-794fe6a925bc/volumes/kubernetes.io~projected/kube-api-access-tdvfj DeviceMajor:0 DeviceMinor:219 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9af8b8a8-47f6-4576-87db-2f0253903e5b/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:243 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5c959caa-7239-4b81-8c19-c867fc6c48e6/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f71460db-658f-45ed-980f-da65509e2bc5/volumes/kubernetes.io~projected/kube-api-access-pzs82 DeviceMajor:0 DeviceMinor:51 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3af15ebe-c67a-40c5-bac0-24410542413b/volumes/kubernetes.io~projected/kube-api-access-nkg5t DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e23e2763a3591380a493e03e4209f502c432d0e813a2fcfb32604c549226ec42/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/54ac1aee-48aa-4f54-ae3c-9d5d4e377e12/volumes/kubernetes.io~projected/kube-api-access-64mk6 DeviceMajor:0 DeviceMinor:224 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3bed4cac-d251-49be-9eb6-ce5cdb6852de/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5c959caa-7239-4b81-8c19-c867fc6c48e6/volumes/kubernetes.io~projected/kube-api-access-s67jv DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f0222f39630c858fec04d8f23a314c7c5687f9e0d827064de1b96d54d4bccdeb/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3eeb8650-24f6-4e09-95be-794fe6a925bc/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:214 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:296 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/caa1cde0c045e60f020bda4fa2b9c477955b5b0f1ed5dda0dc33c7b8df8916b6/userdata/shm DeviceMajor:0 DeviceMinor:58 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/af19768f02c54b9ae643154322ededc99c4bbbb36ffce3831f3930c16bc8fdc0/userdata/shm DeviceMajor:0 DeviceMinor:257 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c048598f70f7855d3d2d4e4bc8cae60c611b1a3384a3de61ef4b6064fe9f6742/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/25bfba8acd1e8775dfcc86ff3f4e202898c47daba1a52956e44466ce2d833b47/userdata/shm DeviceMajor:0 DeviceMinor:255 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:01c5e91dacb9419 MacAddress:8e:61:70:e2:d1:3b Speed:10000 Mtu:8901} {Name:25bfba8acd1e877 MacAddress:8a:e1:b2:07:eb:20 Speed:10000 Mtu:8901} {Name:46c075f064de3dc MacAddress:0e:05:f7:13:bb:ae Speed:10000 Mtu:8901} {Name:5cf72dade1becca MacAddress:46:06:3a:1d:37:b1 Speed:10000 Mtu:8901} {Name:7385d8d21508890 MacAddress:2a:60:33:21:44:65 Speed:10000 Mtu:8901} {Name:9e5f098e5bd56bb MacAddress:c6:02:60:29:d8:56 Speed:10000 Mtu:8901} {Name:af19768f02c54b9 MacAddress:1e:ea:5b:66:1c:4d Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d1:b0:d4:1d Speed:0 Mtu:9001} {Name:br-int MacAddress:5a:17:e1:b8:fd:11 Speed:0 Mtu:8901} {Name:c048598f70f7855 MacAddress:02:24:ab:8d:5f:40 Speed:10000 Mtu:8901} {Name:ca29d6a9e0c4f4a MacAddress:46:05:2e:2e:bd:27 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d1:b0:d4:1d Speed:0 Mtu:9001} {Name:f893907a88ffd76 MacAddress:ea:97:9f:77:2c:8e Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:4a:de:7e:86:aa:d5 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:16:eb:6e:7e:40:99 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 22:23:51.663887 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.661459 5011 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 22:23:51.663887 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.661499 5011 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 22:23:51.663887 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.661812 5011 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 22:23:51.663887 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.661852 5011 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-238.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 22:23:51.663887 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.662021 5011 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 22:23:51.663887 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.662029 5011 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 22:23:51.663887 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.662043 5011 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 22:23:51.663887 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.662056 5011 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 22:23:51.663887 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.662149 5011 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 22:23:51.663887 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.662326 5011 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 22:23:51.663887 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.662434 5011 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 22:23:51.663887 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.662471 5011 kubelet.go:493] "Attempting to sync node with API server" Nov 01 22:23:51.663887 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.662486 5011 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 22:23:51.663887 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.662496 5011 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 22:23:51.663887 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.662515 5011 kubelet.go:405] "Adding apiserver pod source" Nov 01 22:23:51.663887 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.662525 5011 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 22:23:51.663887 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.663578 5011 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 22:23:51.663887 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.663591 5011 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 22:23:51.664612 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.664187 5011 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 22:23:51.676031 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.676010 5011 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:23:51.677625 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.677595 5011 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:23:51.698614 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.698590 5011 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 22:23:51.698614 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.698606 5011 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 22:23:51.698761 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.698725 5011 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 01 22:23:51.698992 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.698976 5011 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 22:23:51.699074 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.699017 5011 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 22:23:51.699157 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.699146 5011 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 22:23:51.699210 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.699165 5011 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 22:23:51.699210 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.699177 5011 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 22:23:51.699210 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.699186 5011 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 22:23:51.699210 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.699195 5011 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 22:23:51.699210 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.699204 5011 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 22:23:51.699432 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.699214 5011 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 22:23:51.699432 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.699222 5011 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 22:23:51.699432 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.699234 5011 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 22:23:51.699432 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.699243 5011 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 22:23:51.699432 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.699255 5011 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 22:23:51.699432 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.699270 5011 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 22:23:51.699432 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.699306 5011 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 22:23:51.699432 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.699316 5011 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 22:23:51.702882 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.702866 5011 server.go:1263] "Started kubelet" Nov 01 22:23:51.703393 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.702992 5011 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 22:23:51.703393 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.703073 5011 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 22:23:51.703393 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.703016 5011 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 22:23:51.703393 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.703370 5011 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 22:23:51.703458 ip-10-0-142-238 systemd[1]: Started Kubernetes Kubelet. Nov 01 22:23:51.704543 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.704526 5011 server.go:317] "Adding debug handlers to kubelet server" Nov 01 22:23:51.708266 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.707649 5011 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 22:23:51.708642 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.708623 5011 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 22:23:51.708970 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.708953 5011 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 22:23:51.708970 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.708971 5011 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 22:23:51.709088 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.709001 5011 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 22:23:51.709148 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.709072 5011 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 22:17:41 +0000 UTC" deadline="2028-08-14 10:45:49.58438046 +0000 UTC" Nov 01 22:23:51.709148 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.709110 5011 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15636h21m57.875274604s" Nov 01 22:23:51.710825 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.710788 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3a3cc56-6cfb-4092-ba39-5c237ad58c77" volumeName="kubernetes.io/secret/b3a3cc56-6cfb-4092-ba39-5c237ad58c77-cert" seLinuxMountContext="" Nov 01 22:23:51.710978 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.710850 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e296d10d-5064-4f8f-85f0-e6d1b20f1ab8" volumeName="kubernetes.io/configmap/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8-metrics-client-ca" seLinuxMountContext="" Nov 01 22:23:51.710978 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.710870 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e296d10d-5064-4f8f-85f0-e6d1b20f1ab8" volumeName="kubernetes.io/secret/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8-prometheus-operator-tls" seLinuxMountContext="" Nov 01 22:23:51.710978 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.710885 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3880904a-615b-44ac-9a2d-7b0f7ad31e0c" volumeName="kubernetes.io/configmap/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-trusted-ca-bundle" seLinuxMountContext="" Nov 01 22:23:51.710978 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.710904 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3af15ebe-c67a-40c5-bac0-24410542413b" volumeName="kubernetes.io/configmap/3af15ebe-c67a-40c5-bac0-24410542413b-cni-sysctl-allowlist" seLinuxMountContext="" Nov 01 22:23:51.710978 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.710917 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c959caa-7239-4b81-8c19-c867fc6c48e6" volumeName="kubernetes.io/projected/5c959caa-7239-4b81-8c19-c867fc6c48e6-kube-api-access-s67jv" seLinuxMountContext="" Nov 01 22:23:51.710978 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.710935 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9af8b8a8-47f6-4576-87db-2f0253903e5b" volumeName="kubernetes.io/configmap/9af8b8a8-47f6-4576-87db-2f0253903e5b-config-volume" seLinuxMountContext="" Nov 01 22:23:51.710978 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.710949 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb7b37b5-9639-45da-b90e-e08dcfac93f3" volumeName="kubernetes.io/configmap/eb7b37b5-9639-45da-b90e-e08dcfac93f3-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 01 22:23:51.710978 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.710978 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f71460db-658f-45ed-980f-da65509e2bc5" volumeName="kubernetes.io/projected/f71460db-658f-45ed-980f-da65509e2bc5-kube-api-access-pzs82" seLinuxMountContext="" Nov 01 22:23:51.711396 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.710992 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa603801-6c30-4cf1-ace8-30c8723e4a1e" volumeName="kubernetes.io/secret/fa603801-6c30-4cf1-ace8-30c8723e4a1e-original-pull-secret" seLinuxMountContext="" Nov 01 22:23:51.711396 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711016 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bed4cac-d251-49be-9eb6-ce5cdb6852de" volumeName="kubernetes.io/configmap/3bed4cac-d251-49be-9eb6-ce5cdb6852de-konnectivity-ca" seLinuxMountContext="" Nov 01 22:23:51.711396 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711035 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c959caa-7239-4b81-8c19-c867fc6c48e6" volumeName="kubernetes.io/empty-dir/5c959caa-7239-4b81-8c19-c867fc6c48e6-tmp" seLinuxMountContext="" Nov 01 22:23:51.711396 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711047 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e296d10d-5064-4f8f-85f0-e6d1b20f1ab8" volumeName="kubernetes.io/secret/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Nov 01 22:23:51.711396 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711066 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb7b37b5-9639-45da-b90e-e08dcfac93f3" volumeName="kubernetes.io/projected/eb7b37b5-9639-45da-b90e-e08dcfac93f3-kube-api-access-vm2vf" seLinuxMountContext="" Nov 01 22:23:51.711396 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711088 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f71460db-658f-45ed-980f-da65509e2bc5" volumeName="kubernetes.io/empty-dir/f71460db-658f-45ed-980f-da65509e2bc5-tmp-dir" seLinuxMountContext="" Nov 01 22:23:51.711396 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711107 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bed4cac-d251-49be-9eb6-ce5cdb6852de" volumeName="kubernetes.io/secret/3bed4cac-d251-49be-9eb6-ce5cdb6852de-agent-certs" seLinuxMountContext="" Nov 01 22:23:51.711396 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711120 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eeb8650-24f6-4e09-95be-794fe6a925bc" volumeName="kubernetes.io/configmap/3eeb8650-24f6-4e09-95be-794fe6a925bc-signing-cabundle" seLinuxMountContext="" Nov 01 22:23:51.711396 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711134 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9af8b8a8-47f6-4576-87db-2f0253903e5b" volumeName="kubernetes.io/projected/9af8b8a8-47f6-4576-87db-2f0253903e5b-kube-api-access-h8wg9" seLinuxMountContext="" Nov 01 22:23:51.711396 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711155 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b439b405-8bca-4ce8-980d-a076cb2f11bf" volumeName="kubernetes.io/projected/b439b405-8bca-4ce8-980d-a076cb2f11bf-kube-api-access-lxhl8" seLinuxMountContext="" Nov 01 22:23:51.711396 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711169 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e296d10d-5064-4f8f-85f0-e6d1b20f1ab8" volumeName="kubernetes.io/projected/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8-kube-api-access-gwr44" seLinuxMountContext="" Nov 01 22:23:51.711396 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711188 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb7b37b5-9639-45da-b90e-e08dcfac93f3" volumeName="kubernetes.io/empty-dir/eb7b37b5-9639-45da-b90e-e08dcfac93f3-data-volume" seLinuxMountContext="" Nov 01 22:23:51.711396 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711203 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14cdd491-7383-4cc5-ac4f-60a75110d764" volumeName="kubernetes.io/projected/14cdd491-7383-4cc5-ac4f-60a75110d764-kube-api-access-n9q9h" seLinuxMountContext="" Nov 01 22:23:51.711396 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711217 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3880904a-615b-44ac-9a2d-7b0f7ad31e0c" volumeName="kubernetes.io/projected/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-kube-api-access-8ndlv" seLinuxMountContext="" Nov 01 22:23:51.711396 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711235 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45d5cfa6-bf31-4fad-8003-d28fe9a37183" volumeName="kubernetes.io/secret/45d5cfa6-bf31-4fad-8003-d28fe9a37183-ovn-node-metrics-cert" seLinuxMountContext="" Nov 01 22:23:51.711396 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711248 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3a3cc56-6cfb-4092-ba39-5c237ad58c77" volumeName="kubernetes.io/projected/b3a3cc56-6cfb-4092-ba39-5c237ad58c77-kube-api-access-qjzrz" seLinuxMountContext="" Nov 01 22:23:51.711396 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711267 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc3de28d-1139-4677-81db-a114c8867933" volumeName="kubernetes.io/projected/cc3de28d-1139-4677-81db-a114c8867933-kube-api-access-6nvbs" seLinuxMountContext="" Nov 01 22:23:51.711396 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711282 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="179a164c-f6dd-42f9-9428-f241f81afff5" volumeName="kubernetes.io/configmap/179a164c-f6dd-42f9-9428-f241f81afff5-multus-daemon-config" seLinuxMountContext="" Nov 01 22:23:51.711396 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711296 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3880904a-615b-44ac-9a2d-7b0f7ad31e0c" volumeName="kubernetes.io/empty-dir/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-archives-path" seLinuxMountContext="" Nov 01 22:23:51.711396 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711320 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3af15ebe-c67a-40c5-bac0-24410542413b" volumeName="kubernetes.io/configmap/3af15ebe-c67a-40c5-bac0-24410542413b-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 01 22:23:51.711396 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711346 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eeb8650-24f6-4e09-95be-794fe6a925bc" volumeName="kubernetes.io/secret/3eeb8650-24f6-4e09-95be-794fe6a925bc-signing-key" seLinuxMountContext="" Nov 01 22:23:51.711396 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711359 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45d5cfa6-bf31-4fad-8003-d28fe9a37183" volumeName="kubernetes.io/configmap/45d5cfa6-bf31-4fad-8003-d28fe9a37183-ovnkube-config" seLinuxMountContext="" Nov 01 22:23:51.711396 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711384 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45d5cfa6-bf31-4fad-8003-d28fe9a37183" volumeName="kubernetes.io/configmap/45d5cfa6-bf31-4fad-8003-d28fe9a37183-ovnkube-script-lib" seLinuxMountContext="" Nov 01 22:23:51.711396 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711405 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45d5cfa6-bf31-4fad-8003-d28fe9a37183" volumeName="kubernetes.io/configmap/45d5cfa6-bf31-4fad-8003-d28fe9a37183-env-overrides" seLinuxMountContext="" Nov 01 22:23:51.712304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711419 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb7b37b5-9639-45da-b90e-e08dcfac93f3" volumeName="kubernetes.io/secret/eb7b37b5-9639-45da-b90e-e08dcfac93f3-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 01 22:23:51.712304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711439 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3af15ebe-c67a-40c5-bac0-24410542413b" volumeName="kubernetes.io/projected/3af15ebe-c67a-40c5-bac0-24410542413b-kube-api-access-nkg5t" seLinuxMountContext="" Nov 01 22:23:51.712304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711452 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9af8b8a8-47f6-4576-87db-2f0253903e5b" volumeName="kubernetes.io/empty-dir/9af8b8a8-47f6-4576-87db-2f0253903e5b-tmp-dir" seLinuxMountContext="" Nov 01 22:23:51.712304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711470 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc3de28d-1139-4677-81db-a114c8867933" volumeName="kubernetes.io/secret/cc3de28d-1139-4677-81db-a114c8867933-metrics-certs" seLinuxMountContext="" Nov 01 22:23:51.712304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711485 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="179a164c-f6dd-42f9-9428-f241f81afff5" volumeName="kubernetes.io/configmap/179a164c-f6dd-42f9-9428-f241f81afff5-cni-binary-copy" seLinuxMountContext="" Nov 01 22:23:51.712304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711512 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="179a164c-f6dd-42f9-9428-f241f81afff5" volumeName="kubernetes.io/projected/179a164c-f6dd-42f9-9428-f241f81afff5-kube-api-access-sstcj" seLinuxMountContext="" Nov 01 22:23:51.712304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711530 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3af15ebe-c67a-40c5-bac0-24410542413b" volumeName="kubernetes.io/configmap/3af15ebe-c67a-40c5-bac0-24410542413b-cni-binary-copy" seLinuxMountContext="" Nov 01 22:23:51.712304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711545 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eeb8650-24f6-4e09-95be-794fe6a925bc" volumeName="kubernetes.io/projected/3eeb8650-24f6-4e09-95be-794fe6a925bc-kube-api-access-tdvfj" seLinuxMountContext="" Nov 01 22:23:51.712304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711562 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8" volumeName="kubernetes.io/configmap/3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8-iptables-alerter-script" seLinuxMountContext="" Nov 01 22:23:51.712304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711576 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45d5cfa6-bf31-4fad-8003-d28fe9a37183" volumeName="kubernetes.io/projected/45d5cfa6-bf31-4fad-8003-d28fe9a37183-kube-api-access-kw4f7" seLinuxMountContext="" Nov 01 22:23:51.712304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711589 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54ac1aee-48aa-4f54-ae3c-9d5d4e377e12" volumeName="kubernetes.io/projected/54ac1aee-48aa-4f54-ae3c-9d5d4e377e12-kube-api-access-64mk6" seLinuxMountContext="" Nov 01 22:23:51.712304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711608 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c959caa-7239-4b81-8c19-c867fc6c48e6" volumeName="kubernetes.io/empty-dir/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-tuned" seLinuxMountContext="" Nov 01 22:23:51.712304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711622 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3880904a-615b-44ac-9a2d-7b0f7ad31e0c" volumeName="kubernetes.io/configmap/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-service-ca-bundle" seLinuxMountContext="" Nov 01 22:23:51.712304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711639 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8" volumeName="kubernetes.io/projected/3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8-kube-api-access-mqg4h" seLinuxMountContext="" Nov 01 22:23:51.712304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711653 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="682b0c5d-fa6f-4a17-8b72-a1f224cf5fec" volumeName="kubernetes.io/configmap/682b0c5d-fa6f-4a17-8b72-a1f224cf5fec-serviceca" seLinuxMountContext="" Nov 01 22:23:51.712304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711666 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="682b0c5d-fa6f-4a17-8b72-a1f224cf5fec" volumeName="kubernetes.io/projected/682b0c5d-fa6f-4a17-8b72-a1f224cf5fec-kube-api-access-fd2dd" seLinuxMountContext="" Nov 01 22:23:51.712304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711683 5011 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9af8b8a8-47f6-4576-87db-2f0253903e5b" volumeName="kubernetes.io/secret/9af8b8a8-47f6-4576-87db-2f0253903e5b-metrics-tls" seLinuxMountContext="" Nov 01 22:23:51.712304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711694 5011 reconstruct.go:97] "Volume reconstruction finished" Nov 01 22:23:51.712304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.711703 5011 reconciler.go:29] "Reconciler: start to sync state" Nov 01 22:23:51.713137 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.712343 5011 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:23:51.713137 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.712543 5011 factory.go:55] Registering systemd factory Nov 01 22:23:51.713137 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.712558 5011 factory.go:223] Registration of the systemd container factory successfully Nov 01 22:23:51.713137 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.712870 5011 factory.go:153] Registering CRI-O factory Nov 01 22:23:51.713137 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.712887 5011 factory.go:223] Registration of the crio container factory successfully Nov 01 22:23:51.713137 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.712942 5011 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 22:23:51.713137 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.712960 5011 factory.go:103] Registering Raw factory Nov 01 22:23:51.713137 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.712974 5011 manager.go:1199] Started watching for new ooms in manager Nov 01 22:23:51.713363 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.713308 5011 manager.go:319] Starting recovery of all containers Nov 01 22:23:51.714812 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.714798 5011 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 01 22:23:51.737307 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.737276 5011 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 22:23:51.738724 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.738685 5011 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 22:23:51.738819 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.738736 5011 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 22:23:51.738819 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.738760 5011 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 22:23:51.738819 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:23:51.738800 5011 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 01 22:23:51.743446 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.743425 5011 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:23:51.773423 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.773355 5011 generic.go:356] "Generic (PLEG): container finished" podID="3af15ebe-c67a-40c5-bac0-24410542413b" containerID="75629e491d6b06c975de9654b36e8e5f27055e05c4ce39087e8e98bec9b88eb5" exitCode=0 Nov 01 22:23:51.773423 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.773382 5011 generic.go:356] "Generic (PLEG): container finished" podID="3af15ebe-c67a-40c5-bac0-24410542413b" containerID="e7dddd97aa426a465a3958cc8bf2636f5c91fc2b102bd9d625d5ef5aa0c3340b" exitCode=0 Nov 01 22:23:51.773423 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.773390 5011 generic.go:356] "Generic (PLEG): container finished" podID="3af15ebe-c67a-40c5-bac0-24410542413b" containerID="a61d238af5ad4bed3be78c4ad6a75ecdd29a2597f5eec7f03e8e1841704ab771" exitCode=0 Nov 01 22:23:51.773423 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.773401 5011 generic.go:356] "Generic (PLEG): container finished" podID="3af15ebe-c67a-40c5-bac0-24410542413b" containerID="119ac3863d9b933467c639e15af6bc4b571e6a2be1ff835169d0a2047ee1945f" exitCode=0 Nov 01 22:23:51.773423 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.773409 5011 generic.go:356] "Generic (PLEG): container finished" podID="3af15ebe-c67a-40c5-bac0-24410542413b" containerID="de4a86630cd89dae0d71ecbd02a0186f63d66dd7865628934190c5fa18d80294" exitCode=0 Nov 01 22:23:51.773423 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.773415 5011 generic.go:356] "Generic (PLEG): container finished" podID="3af15ebe-c67a-40c5-bac0-24410542413b" containerID="687214760f8a2c4c662547487e75245b6a52aa09487e8a84f4644c7257c5e5ef" exitCode=0 Nov 01 22:23:51.775274 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.775240 5011 generic.go:356] "Generic (PLEG): container finished" podID="fbc5f96efa48930e21a66fef652c693e" containerID="6d07763750c179a74435723f3dd52c79ce6ba457ef0de02540c5b793d0a7fcca" exitCode=0 Nov 01 22:23:51.795139 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.795111 5011 manager.go:324] Recovery completed Nov 01 22:23:51.813300 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.813273 5011 cpu_manager.go:226] "Starting" policy="none" Nov 01 22:23:51.813300 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.813290 5011 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 22:23:51.813478 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.813310 5011 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 22:23:51.813478 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.813450 5011 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 01 22:23:51.813478 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.813464 5011 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 01 22:23:51.813637 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.813531 5011 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 01 22:23:51.813637 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.813542 5011 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 01 22:23:51.813637 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.813550 5011 policy_none.go:50] "Start" Nov 01 22:23:51.813637 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.813572 5011 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 22:23:51.813637 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.813585 5011 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 22:23:51.813872 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.813769 5011 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 01 22:23:51.813872 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.813780 5011 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 01 22:23:51.813872 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.813788 5011 policy_none.go:44] "Start" Nov 01 22:23:51.818428 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.817980 5011 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 22:23:51.823457 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.823445 5011 manager.go:341] "Starting Device Plugin manager" Nov 01 22:23:51.823511 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:23:51.823470 5011 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 22:23:51.823511 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.823479 5011 server.go:87] "Starting device plugin registration server" Nov 01 22:23:51.823752 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.823739 5011 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 22:23:51.823795 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.823754 5011 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 22:23:51.823869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.823849 5011 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 22:23:51.823977 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.823935 5011 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 22:23:51.823977 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.823974 5011 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 22:23:51.824094 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.823981 5011 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 22:23:51.824094 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.824022 5011 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T22:23:51.823964933Z","UUID":"85b5b8b6-a9ca-409e-9061-0fcc3e69ddc1","Handler":null,"Name":"","Endpoint":""} Nov 01 22:23:51.825377 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:23:51.825353 5011 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 22:23:51.826532 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.826461 5011 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 22:23:51.826532 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.826529 5011 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 22:23:51.838924 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.838873 5011 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-238.us-east-2.compute.internal"] Nov 01 22:23:51.839601 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.839586 5011 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:23:51.839684 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.839644 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal" event={"ID":"fbc5f96efa48930e21a66fef652c693e","Type":"ContainerStarted","Data":"687172ba2729d825e69e7fea1c2fa891ae7c7810ddec386070b79393dcf4182d"} Nov 01 22:23:51.839740 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.839700 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal" event={"ID":"fbc5f96efa48930e21a66fef652c693e","Type":"ContainerDied","Data":"6d07763750c179a74435723f3dd52c79ce6ba457ef0de02540c5b793d0a7fcca"} Nov 01 22:23:51.839740 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.839721 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal" event={"ID":"fbc5f96efa48930e21a66fef652c693e","Type":"ContainerStarted","Data":"1d8b4c5a4a30cc4854e7d670f40126715432e1fba59c5f74cf836857393e1784"} Nov 01 22:23:51.840018 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.839766 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-238.us-east-2.compute.internal" event={"ID":"4f1e77a94f837e0f1b12b6a1e77eab89","Type":"ContainerStarted","Data":"8d65edf964dbf913fbee970830bb966eb40d9348b49138573cc5fd5f1a457395"} Nov 01 22:23:51.840018 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.839786 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-238.us-east-2.compute.internal" event={"ID":"4f1e77a94f837e0f1b12b6a1e77eab89","Type":"ContainerStarted","Data":"caa1cde0c045e60f020bda4fa2b9c477955b5b0f1ed5dda0dc33c7b8df8916b6"} Nov 01 22:23:51.840018 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.839874 5011 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:23:51.851730 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.851710 5011 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 22:23:51.851870 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:23:51.851755 5011 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-238.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:23:51.851925 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.851891 5011 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 22:23:51.851968 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:23:51.851922 5011 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:23:51.915775 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.915742 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fbc5f96efa48930e21a66fef652c693e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal\" (UID: \"fbc5f96efa48930e21a66fef652c693e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:23:51.915775 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.915775 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbc5f96efa48930e21a66fef652c693e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal\" (UID: \"fbc5f96efa48930e21a66fef652c693e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:23:51.915973 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.915796 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4f1e77a94f837e0f1b12b6a1e77eab89-config\") pod \"kube-apiserver-proxy-ip-10-0-142-238.us-east-2.compute.internal\" (UID: \"4f1e77a94f837e0f1b12b6a1e77eab89\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:23:51.924867 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.924848 5011 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:23:51.928286 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.928269 5011 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-238.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:23:51.928366 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.928301 5011 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-238.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:23:51.928366 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.928313 5011 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-238.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:23:51.928366 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.928350 5011 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:23:51.940802 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.940784 5011 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:23:51.940898 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:51.940854 5011 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:23:52.016337 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.016249 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbc5f96efa48930e21a66fef652c693e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal\" (UID: \"fbc5f96efa48930e21a66fef652c693e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:23:52.016492 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.016361 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4f1e77a94f837e0f1b12b6a1e77eab89-config\") pod \"kube-apiserver-proxy-ip-10-0-142-238.us-east-2.compute.internal\" (UID: \"4f1e77a94f837e0f1b12b6a1e77eab89\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:23:52.016492 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.016382 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fbc5f96efa48930e21a66fef652c693e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal\" (UID: \"fbc5f96efa48930e21a66fef652c693e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:23:52.016492 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.016309 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbc5f96efa48930e21a66fef652c693e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal\" (UID: \"fbc5f96efa48930e21a66fef652c693e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:23:52.016492 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.016421 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fbc5f96efa48930e21a66fef652c693e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal\" (UID: \"fbc5f96efa48930e21a66fef652c693e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:23:52.016492 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.016442 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4f1e77a94f837e0f1b12b6a1e77eab89-config\") pod \"kube-apiserver-proxy-ip-10-0-142-238.us-east-2.compute.internal\" (UID: \"4f1e77a94f837e0f1b12b6a1e77eab89\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:23:52.663546 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.663512 5011 apiserver.go:52] "Watching apiserver" Nov 01 22:23:52.673653 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.673627 5011 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 22:23:52.674137 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.674117 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-ttjsz","openshift-image-registry/node-ca-rkf2j","openshift-kube-storage-version-migrator/migrator-66df5d496b-9s6vz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr","openshift-insights/insights-runtime-extractor-lcqwg","openshift-ovn-kubernetes/ovnkube-node-6h899","kube-system/konnectivity-agent-tmhm2","openshift-monitoring/prometheus-operator-9cd89b74c-sxspm","openshift-ingress-canary/ingress-canary-v2l2j","openshift-network-operator/iptables-alerter-npsp2","openshift-service-ca/service-ca-5fffc97bdf-ss7dr","openshift-multus/multus-additional-cni-plugins-jngxk","openshift-cluster-node-tuning-operator/tuned-mhvjj","openshift-multus/multus-84vvx","openshift-dns/dns-default-srdnl","kube-system/kube-apiserver-proxy-ip-10-0-142-238.us-east-2.compute.internal","openshift-insights/periodic-gathering-sk8f6-sf7bv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal","openshift-network-diagnostics/network-check-target-cfnmp","openshift-multus/network-metrics-daemon-xvgrl","openshift-dns/node-resolver-x4k8b"] Nov 01 22:23:52.679057 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.679037 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 22:23:52.679214 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.679189 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 22:23:52.679306 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.679243 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 22:23:52.679306 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.679260 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qzblw\"" Nov 01 22:23:52.679306 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.679260 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jvq4g\"" Nov 01 22:23:52.680448 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.680430 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z74s5\"" Nov 01 22:23:52.680599 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.680584 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 22:23:52.680668 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.680630 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 22:23:52.680668 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.680642 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xdns4\"" Nov 01 22:23:52.680777 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.680677 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 22:23:52.680777 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.680699 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 22:23:52.680777 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.680730 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 22:23:52.680956 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.680938 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 22:23:52.680956 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.680950 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 22:23:52.681060 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.680962 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 22:23:52.681060 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.680948 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 22:23:52.681060 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.680970 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 22:23:52.681590 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.681558 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 22:23:52.681670 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.681608 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:23:52.682168 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.682156 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 22:23:52.682221 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.682198 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mvk6z\"" Nov 01 22:23:52.682326 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.682311 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nz8z8\"" Nov 01 22:23:52.682404 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.682389 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sml6h\"" Nov 01 22:23:52.683323 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.683306 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 22:23:52.683573 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.683555 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 22:23:52.683647 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.683634 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sw8fs\"" Nov 01 22:23:52.684181 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.684162 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 22:23:52.684270 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.684259 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5sq9j\"" Nov 01 22:23:52.684463 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.684438 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:23:52.684595 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.684583 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 22:23:52.684664 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.684638 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 22:23:52.684900 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.684880 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6vwc2\"" Nov 01 22:23:52.684985 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.684970 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 22:23:52.684985 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.684974 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 22:23:52.685067 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.684992 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 22:23:52.685159 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.685145 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 22:23:52.685159 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.685154 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9gzz5\"" Nov 01 22:23:52.685243 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.685200 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 22:23:52.685449 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.685433 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 22:23:52.685505 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.685489 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 22:23:52.685591 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.685531 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 22:23:52.685689 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.685676 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 22:23:52.685721 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.685684 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 22:23:52.686209 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.686196 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 22:23:52.686539 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.686519 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 22:23:52.686609 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.686547 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-th659\"" Nov 01 22:23:52.686609 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.686547 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 22:23:52.686609 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.686548 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fdq7t\"" Nov 01 22:23:52.686609 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.686604 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 22:23:52.686774 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.686692 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hhd78\"" Nov 01 22:23:52.687012 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.686994 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 22:23:52.687012 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.687004 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 22:23:52.687132 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.687032 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 22:23:52.687132 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.687034 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l7bq7\"" Nov 01 22:23:52.687132 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.687090 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 22:23:52.687377 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.687363 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-gz5rr\"" Nov 01 22:23:52.687619 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.687608 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 01 22:23:52.688266 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.688248 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 01 22:23:52.688266 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.688261 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-5lf4f\"" Nov 01 22:23:52.688266 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.688262 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 01 22:23:52.688424 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.688257 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 01 22:23:52.688424 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.688369 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 01 22:23:52.689954 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.689937 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-d6jqb\"" Nov 01 22:23:52.689954 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.689943 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 01 22:23:52.690379 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.690363 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 01 22:23:52.691459 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.691441 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 22:23:52.691532 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.691492 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 22:23:52.691532 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.691499 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 01 22:23:52.691678 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.691665 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 01 22:23:52.691717 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.691668 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 22:23:52.691717 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.691680 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-rrpdg\"" Nov 01 22:23:52.692337 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.692323 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 01 22:23:52.710423 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.710402 5011 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 22:23:52.718598 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.718575 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdvfj\" (UniqueName: \"kubernetes.io/projected/3eeb8650-24f6-4e09-95be-794fe6a925bc-kube-api-access-tdvfj\") pod \"service-ca-5fffc97bdf-ss7dr\" (UID: \"3eeb8650-24f6-4e09-95be-794fe6a925bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ss7dr" Nov 01 22:23:52.718688 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.718606 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-sysconfig\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.718688 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.718629 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-var-lib-cni-bin\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.718688 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.718669 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-service-ca-bundle\") pod \"periodic-gathering-sk8f6-sf7bv\" (UID: \"3880904a-615b-44ac-9a2d-7b0f7ad31e0c\") " pod="openshift-insights/periodic-gathering-sk8f6-sf7bv" Nov 01 22:23:52.718891 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.718750 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-sxspm\" (UID: \"e296d10d-5064-4f8f-85f0-e6d1b20f1ab8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-sxspm" Nov 01 22:23:52.718969 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.718932 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3af15ebe-c67a-40c5-bac0-24410542413b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:23:52.719065 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.719010 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-run-openvswitch\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.719065 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.719028 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-sxspm\" (UID: \"e296d10d-5064-4f8f-85f0-e6d1b20f1ab8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-sxspm" Nov 01 22:23:52.719065 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.719047 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-sysctl-d\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.719184 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.719071 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-multus-cni-dir\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.719184 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.719112 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb7b37b5-9639-45da-b90e-e08dcfac93f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:52.719184 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.719158 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3af15ebe-c67a-40c5-bac0-24410542413b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:23:52.719317 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.719243 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3af15ebe-c67a-40c5-bac0-24410542413b-cnibin\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:23:52.719317 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.719280 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-run-ovn-kubernetes\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.719317 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.719307 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-service-ca-bundle\") pod \"periodic-gathering-sk8f6-sf7bv\" (UID: \"3880904a-615b-44ac-9a2d-7b0f7ad31e0c\") " pod="openshift-insights/periodic-gathering-sk8f6-sf7bv" Nov 01 22:23:52.719495 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.719310 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/45d5cfa6-bf31-4fad-8003-d28fe9a37183-ovn-node-metrics-cert\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.719495 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.719349 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/45d5cfa6-bf31-4fad-8003-d28fe9a37183-ovnkube-script-lib\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.719495 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.719373 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-modprobe-d\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.719495 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.719396 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-system-cni-dir\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.719495 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.719418 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-var-lib-kubelet\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.719495 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.719426 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb7b37b5-9639-45da-b90e-e08dcfac93f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:52.719495 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.719467 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f71460db-658f-45ed-980f-da65509e2bc5-tmp-dir\") pod \"node-resolver-x4k8b\" (UID: \"f71460db-658f-45ed-980f-da65509e2bc5\") " pod="openshift-dns/node-resolver-x4k8b" Nov 01 22:23:52.719793 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.719570 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/45d5cfa6-bf31-4fad-8003-d28fe9a37183-ovn-node-metrics-cert\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.719793 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.719578 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzs82\" (UniqueName: \"kubernetes.io/projected/f71460db-658f-45ed-980f-da65509e2bc5-kube-api-access-pzs82\") pod \"node-resolver-x4k8b\" (UID: \"f71460db-658f-45ed-980f-da65509e2bc5\") " pod="openshift-dns/node-resolver-x4k8b" Nov 01 22:23:52.719793 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.719609 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f71460db-658f-45ed-980f-da65509e2bc5-tmp-dir\") pod \"node-resolver-x4k8b\" (UID: \"f71460db-658f-45ed-980f-da65509e2bc5\") " pod="openshift-dns/node-resolver-x4k8b" Nov 01 22:23:52.719793 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.719614 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-sys-fs\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:23:52.719793 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.719766 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-kubelet\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.719793 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.719774 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/45d5cfa6-bf31-4fad-8003-d28fe9a37183-ovnkube-script-lib\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.720117 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.719802 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-run-systemd\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.720117 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.719873 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-lib-modules\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.720117 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.719905 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8-host-slash\") pod \"iptables-alerter-npsp2\" (UID: \"3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8\") " pod="openshift-network-operator/iptables-alerter-npsp2" Nov 01 22:23:52.720117 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.719936 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-etc-kubernetes\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.720117 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.719955 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjzrz\" (UniqueName: \"kubernetes.io/projected/b3a3cc56-6cfb-4092-ba39-5c237ad58c77-kube-api-access-qjzrz\") pod \"ingress-canary-v2l2j\" (UID: \"b3a3cc56-6cfb-4092-ba39-5c237ad58c77\") " pod="openshift-ingress-canary/ingress-canary-v2l2j" Nov 01 22:23:52.720117 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.719971 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxhl8\" (UniqueName: \"kubernetes.io/projected/b439b405-8bca-4ce8-980d-a076cb2f11bf-kube-api-access-lxhl8\") pod \"network-check-target-cfnmp\" (UID: \"b439b405-8bca-4ce8-980d-a076cb2f11bf\") " pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:23:52.720117 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.719986 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-etc-openvswitch\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.720117 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.720003 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/45d5cfa6-bf31-4fad-8003-d28fe9a37183-ovnkube-config\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.720117 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.720017 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/45d5cfa6-bf31-4fad-8003-d28fe9a37183-env-overrides\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.720492 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.720127 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqg4h\" (UniqueName: \"kubernetes.io/projected/3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8-kube-api-access-mqg4h\") pod \"iptables-alerter-npsp2\" (UID: \"3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8\") " pod="openshift-network-operator/iptables-alerter-npsp2" Nov 01 22:23:52.720492 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.720228 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/45d5cfa6-bf31-4fad-8003-d28fe9a37183-env-overrides\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.720492 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.720243 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-run-netns\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.720492 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.720281 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb7b37b5-9639-45da-b90e-e08dcfac93f3-data-volume\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:52.720492 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.720250 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/45d5cfa6-bf31-4fad-8003-d28fe9a37183-ovnkube-config\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.720492 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.720335 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-archives-path\") pod \"periodic-gathering-sk8f6-sf7bv\" (UID: \"3880904a-615b-44ac-9a2d-7b0f7ad31e0c\") " pod="openshift-insights/periodic-gathering-sk8f6-sf7bv" Nov 01 22:23:52.720492 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.720358 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb7b37b5-9639-45da-b90e-e08dcfac93f3-data-volume\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:52.720492 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.720366 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-var-lib-openvswitch\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.720492 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.720394 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw4f7\" (UniqueName: \"kubernetes.io/projected/45d5cfa6-bf31-4fad-8003-d28fe9a37183-kube-api-access-kw4f7\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.720492 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.720408 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-archives-path\") pod \"periodic-gathering-sk8f6-sf7bv\" (UID: \"3880904a-615b-44ac-9a2d-7b0f7ad31e0c\") " pod="openshift-insights/periodic-gathering-sk8f6-sf7bv" Nov 01 22:23:52.720492 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.720437 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9af8b8a8-47f6-4576-87db-2f0253903e5b-config-volume\") pod \"dns-default-srdnl\" (UID: \"9af8b8a8-47f6-4576-87db-2f0253903e5b\") " pod="openshift-dns/dns-default-srdnl" Nov 01 22:23:52.721025 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.720572 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9af8b8a8-47f6-4576-87db-2f0253903e5b-metrics-tls\") pod \"dns-default-srdnl\" (UID: \"9af8b8a8-47f6-4576-87db-2f0253903e5b\") " pod="openshift-dns/dns-default-srdnl" Nov 01 22:23:52.721025 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.720601 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64mk6\" (UniqueName: \"kubernetes.io/projected/54ac1aee-48aa-4f54-ae3c-9d5d4e377e12-kube-api-access-64mk6\") pod \"migrator-66df5d496b-9s6vz\" (UID: \"54ac1aee-48aa-4f54-ae3c-9d5d4e377e12\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9s6vz" Nov 01 22:23:52.721025 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.720626 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-kubernetes\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.721025 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.720652 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkg5t\" (UniqueName: \"kubernetes.io/projected/3af15ebe-c67a-40c5-bac0-24410542413b-kube-api-access-nkg5t\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:23:52.721025 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.720646 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9af8b8a8-47f6-4576-87db-2f0253903e5b-config-volume\") pod \"dns-default-srdnl\" (UID: \"9af8b8a8-47f6-4576-87db-2f0253903e5b\") " pod="openshift-dns/dns-default-srdnl" Nov 01 22:23:52.721025 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.720773 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-slash\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.721025 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.720805 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-cni-netd\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.721025 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.720877 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-systemd\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.721025 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.720903 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9af8b8a8-47f6-4576-87db-2f0253903e5b-metrics-tls\") pod \"dns-default-srdnl\" (UID: \"9af8b8a8-47f6-4576-87db-2f0253903e5b\") " pod="openshift-dns/dns-default-srdnl" Nov 01 22:23:52.721025 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.720931 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-sys\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.721025 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.720980 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-run-k8s-cni-cncf-io\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.721025 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721013 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/682b0c5d-fa6f-4a17-8b72-a1f224cf5fec-serviceca\") pod \"node-ca-rkf2j\" (UID: \"682b0c5d-fa6f-4a17-8b72-a1f224cf5fec\") " pod="openshift-image-registry/node-ca-rkf2j" Nov 01 22:23:52.721593 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721037 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-sxspm\" (UID: \"e296d10d-5064-4f8f-85f0-e6d1b20f1ab8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-sxspm" Nov 01 22:23:52.721593 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721062 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9af8b8a8-47f6-4576-87db-2f0253903e5b-tmp-dir\") pod \"dns-default-srdnl\" (UID: \"9af8b8a8-47f6-4576-87db-2f0253903e5b\") " pod="openshift-dns/dns-default-srdnl" Nov 01 22:23:52.721593 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721087 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc3de28d-1139-4677-81db-a114c8867933-metrics-certs\") pod \"network-metrics-daemon-xvgrl\" (UID: \"cc3de28d-1139-4677-81db-a114c8867933\") " pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:23:52.721593 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721160 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9af8b8a8-47f6-4576-87db-2f0253903e5b-tmp-dir\") pod \"dns-default-srdnl\" (UID: \"9af8b8a8-47f6-4576-87db-2f0253903e5b\") " pod="openshift-dns/dns-default-srdnl" Nov 01 22:23:52.721593 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721185 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-sysctl-conf\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.721593 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721218 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-sxspm\" (UID: \"e296d10d-5064-4f8f-85f0-e6d1b20f1ab8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-sxspm" Nov 01 22:23:52.721593 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721243 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:23:52.721593 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721256 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-sxspm\" (UID: \"e296d10d-5064-4f8f-85f0-e6d1b20f1ab8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-sxspm" Nov 01 22:23:52.721593 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721257 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/682b0c5d-fa6f-4a17-8b72-a1f224cf5fec-serviceca\") pod \"node-ca-rkf2j\" (UID: \"682b0c5d-fa6f-4a17-8b72-a1f224cf5fec\") " pod="openshift-image-registry/node-ca-rkf2j" Nov 01 22:23:52.721593 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721282 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-run-ovn\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.721593 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721339 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc3de28d-1139-4677-81db-a114c8867933-metrics-certs\") pod \"network-metrics-daemon-xvgrl\" (UID: \"cc3de28d-1139-4677-81db-a114c8867933\") " pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:23:52.721593 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721367 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c959caa-7239-4b81-8c19-c867fc6c48e6-tmp\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.721593 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721436 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-sxspm\" (UID: \"e296d10d-5064-4f8f-85f0-e6d1b20f1ab8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-sxspm" Nov 01 22:23:52.721593 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721448 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c959caa-7239-4b81-8c19-c867fc6c48e6-tmp\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.721593 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721465 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-cnibin\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.721593 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721481 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-multus-conf-dir\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.721593 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721503 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sstcj\" (UniqueName: \"kubernetes.io/projected/179a164c-f6dd-42f9-9428-f241f81afff5-kube-api-access-sstcj\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.721593 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721517 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3a3cc56-6cfb-4092-ba39-5c237ad58c77-cert\") pod \"ingress-canary-v2l2j\" (UID: \"b3a3cc56-6cfb-4092-ba39-5c237ad58c77\") " pod="openshift-ingress-canary/ingress-canary-v2l2j" Nov 01 22:23:52.721593 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721531 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-registration-dir\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:23:52.721593 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721587 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fa603801-6c30-4cf1-ace8-30c8723e4a1e-dbus\") pod \"global-pull-secret-syncer-ttjsz\" (UID: \"fa603801-6c30-4cf1-ace8-30c8723e4a1e\") " pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721704 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa603801-6c30-4cf1-ace8-30c8723e4a1e-original-pull-secret\") pod \"global-pull-secret-syncer-ttjsz\" (UID: \"fa603801-6c30-4cf1-ace8-30c8723e4a1e\") " pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721734 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3eeb8650-24f6-4e09-95be-794fe6a925bc-signing-cabundle\") pod \"service-ca-5fffc97bdf-ss7dr\" (UID: \"3eeb8650-24f6-4e09-95be-794fe6a925bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ss7dr" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721743 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3a3cc56-6cfb-4092-ba39-5c237ad58c77-cert\") pod \"ingress-canary-v2l2j\" (UID: \"b3a3cc56-6cfb-4092-ba39-5c237ad58c77\") " pod="openshift-ingress-canary/ingress-canary-v2l2j" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721763 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-run-multus-certs\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721792 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb7b37b5-9639-45da-b90e-e08dcfac93f3-crio-socket\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721823 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ndlv\" (UniqueName: \"kubernetes.io/projected/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-kube-api-access-8ndlv\") pod \"periodic-gathering-sk8f6-sf7bv\" (UID: \"3880904a-615b-44ac-9a2d-7b0f7ad31e0c\") " pod="openshift-insights/periodic-gathering-sk8f6-sf7bv" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721876 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd2dd\" (UniqueName: \"kubernetes.io/projected/682b0c5d-fa6f-4a17-8b72-a1f224cf5fec-kube-api-access-fd2dd\") pod \"node-ca-rkf2j\" (UID: \"682b0c5d-fa6f-4a17-8b72-a1f224cf5fec\") " pod="openshift-image-registry/node-ca-rkf2j" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721906 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3bed4cac-d251-49be-9eb6-ce5cdb6852de-konnectivity-ca\") pod \"konnectivity-agent-tmhm2\" (UID: \"3bed4cac-d251-49be-9eb6-ce5cdb6852de\") " pod="kube-system/konnectivity-agent-tmhm2" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721933 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nvbs\" (UniqueName: \"kubernetes.io/projected/cc3de28d-1139-4677-81db-a114c8867933-kube-api-access-6nvbs\") pod \"network-metrics-daemon-xvgrl\" (UID: \"cc3de28d-1139-4677-81db-a114c8867933\") " pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721937 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3eeb8650-24f6-4e09-95be-794fe6a925bc-signing-cabundle\") pod \"service-ca-5fffc97bdf-ss7dr\" (UID: \"3eeb8650-24f6-4e09-95be-794fe6a925bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ss7dr" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721961 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3eeb8650-24f6-4e09-95be-794fe6a925bc-signing-key\") pod \"service-ca-5fffc97bdf-ss7dr\" (UID: \"3eeb8650-24f6-4e09-95be-794fe6a925bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ss7dr" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.721999 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/179a164c-f6dd-42f9-9428-f241f81afff5-multus-daemon-config\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.722008 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa603801-6c30-4cf1-ace8-30c8723e4a1e-original-pull-secret\") pod \"global-pull-secret-syncer-ttjsz\" (UID: \"fa603801-6c30-4cf1-ace8-30c8723e4a1e\") " pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.722030 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb7b37b5-9639-45da-b90e-e08dcfac93f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.722076 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3af15ebe-c67a-40c5-bac0-24410542413b-system-cni-dir\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.722104 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3af15ebe-c67a-40c5-bac0-24410542413b-cni-binary-copy\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.722146 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3af15ebe-c67a-40c5-bac0-24410542413b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.722175 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-etc-selinux\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.722203 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-host\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.722217 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3eeb8650-24f6-4e09-95be-794fe6a925bc-signing-key\") pod \"service-ca-5fffc97bdf-ss7dr\" (UID: \"3eeb8650-24f6-4e09-95be-794fe6a925bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ss7dr" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.722227 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-tuned\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.722254 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/179a164c-f6dd-42f9-9428-f241f81afff5-multus-daemon-config\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.722263 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm2vf\" (UniqueName: \"kubernetes.io/projected/eb7b37b5-9639-45da-b90e-e08dcfac93f3-kube-api-access-vm2vf\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.722296 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-tuned\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.722296 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3bed4cac-d251-49be-9eb6-ce5cdb6852de-konnectivity-ca\") pod \"konnectivity-agent-tmhm2\" (UID: \"3bed4cac-d251-49be-9eb6-ce5cdb6852de\") " pod="kube-system/konnectivity-agent-tmhm2" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.722372 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-socket-dir\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.722423 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3af15ebe-c67a-40c5-bac0-24410542413b-cni-binary-copy\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.722483 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-systemd-units\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.722513 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb7b37b5-9639-45da-b90e-e08dcfac93f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.722514 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3af15ebe-c67a-40c5-bac0-24410542413b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.722555 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-cni-bin\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.722572 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8wg9\" (UniqueName: \"kubernetes.io/projected/9af8b8a8-47f6-4576-87db-2f0253903e5b-kube-api-access-h8wg9\") pod \"dns-default-srdnl\" (UID: \"9af8b8a8-47f6-4576-87db-2f0253903e5b\") " pod="openshift-dns/dns-default-srdnl" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.722602 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s67jv\" (UniqueName: \"kubernetes.io/projected/5c959caa-7239-4b81-8c19-c867fc6c48e6-kube-api-access-s67jv\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.722626 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-var-lib-cni-multus\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.722666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.722640 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3af15ebe-c67a-40c5-bac0-24410542413b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:23:52.723658 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.722780 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-log-socket\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.723658 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.722807 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fa603801-6c30-4cf1-ace8-30c8723e4a1e-kubelet-config\") pod \"global-pull-secret-syncer-ttjsz\" (UID: \"fa603801-6c30-4cf1-ace8-30c8723e4a1e\") " pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:23:52.723658 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.722852 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-run\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.723658 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.722888 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-var-lib-kubelet\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.723658 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.722924 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/179a164c-f6dd-42f9-9428-f241f81afff5-cni-binary-copy\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.723658 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.722962 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-trusted-ca-bundle\") pod \"periodic-gathering-sk8f6-sf7bv\" (UID: \"3880904a-615b-44ac-9a2d-7b0f7ad31e0c\") " pod="openshift-insights/periodic-gathering-sk8f6-sf7bv" Nov 01 22:23:52.723658 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.723000 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3bed4cac-d251-49be-9eb6-ce5cdb6852de-agent-certs\") pod \"konnectivity-agent-tmhm2\" (UID: \"3bed4cac-d251-49be-9eb6-ce5cdb6852de\") " pod="kube-system/konnectivity-agent-tmhm2" Nov 01 22:23:52.723658 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.723025 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-device-dir\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:23:52.723658 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.723050 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8-iptables-alerter-script\") pod \"iptables-alerter-npsp2\" (UID: \"3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8\") " pod="openshift-network-operator/iptables-alerter-npsp2" Nov 01 22:23:52.723658 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.723077 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-multus-socket-dir-parent\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.723658 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.723108 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f71460db-658f-45ed-980f-da65509e2bc5-hosts-file\") pod \"node-resolver-x4k8b\" (UID: \"f71460db-658f-45ed-980f-da65509e2bc5\") " pod="openshift-dns/node-resolver-x4k8b" Nov 01 22:23:52.723658 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.723121 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/179a164c-f6dd-42f9-9428-f241f81afff5-cni-binary-copy\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.723658 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.723135 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9q9h\" (UniqueName: \"kubernetes.io/projected/14cdd491-7383-4cc5-ac4f-60a75110d764-kube-api-access-n9q9h\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:23:52.723658 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.723160 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.723658 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.723342 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8-iptables-alerter-script\") pod \"iptables-alerter-npsp2\" (UID: \"3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8\") " pod="openshift-network-operator/iptables-alerter-npsp2" Nov 01 22:23:52.723658 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.723435 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-trusted-ca-bundle\") pod \"periodic-gathering-sk8f6-sf7bv\" (UID: \"3880904a-615b-44ac-9a2d-7b0f7ad31e0c\") " pod="openshift-insights/periodic-gathering-sk8f6-sf7bv" Nov 01 22:23:52.723658 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.723483 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-hostroot\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.723658 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.723508 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/682b0c5d-fa6f-4a17-8b72-a1f224cf5fec-host\") pod \"node-ca-rkf2j\" (UID: \"682b0c5d-fa6f-4a17-8b72-a1f224cf5fec\") " pod="openshift-image-registry/node-ca-rkf2j" Nov 01 22:23:52.723658 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.723536 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwr44\" (UniqueName: \"kubernetes.io/projected/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8-kube-api-access-gwr44\") pod \"prometheus-operator-9cd89b74c-sxspm\" (UID: \"e296d10d-5064-4f8f-85f0-e6d1b20f1ab8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-sxspm" Nov 01 22:23:52.723658 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.723583 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-run-netns\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.723658 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.723606 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-node-log\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.724267 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.723863 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3bed4cac-d251-49be-9eb6-ce5cdb6852de-agent-certs\") pod \"konnectivity-agent-tmhm2\" (UID: \"3bed4cac-d251-49be-9eb6-ce5cdb6852de\") " pod="kube-system/konnectivity-agent-tmhm2" Nov 01 22:23:52.724652 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.724636 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdvfj\" (UniqueName: \"kubernetes.io/projected/3eeb8650-24f6-4e09-95be-794fe6a925bc-kube-api-access-tdvfj\") pod \"service-ca-5fffc97bdf-ss7dr\" (UID: \"3eeb8650-24f6-4e09-95be-794fe6a925bc\") " pod="openshift-service-ca/service-ca-5fffc97bdf-ss7dr" Nov 01 22:23:52.725842 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.725810 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzs82\" (UniqueName: \"kubernetes.io/projected/f71460db-658f-45ed-980f-da65509e2bc5-kube-api-access-pzs82\") pod \"node-resolver-x4k8b\" (UID: \"f71460db-658f-45ed-980f-da65509e2bc5\") " pod="openshift-dns/node-resolver-x4k8b" Nov 01 22:23:52.728071 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.728042 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64mk6\" (UniqueName: \"kubernetes.io/projected/54ac1aee-48aa-4f54-ae3c-9d5d4e377e12-kube-api-access-64mk6\") pod \"migrator-66df5d496b-9s6vz\" (UID: \"54ac1aee-48aa-4f54-ae3c-9d5d4e377e12\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9s6vz" Nov 01 22:23:52.728199 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.728162 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw4f7\" (UniqueName: \"kubernetes.io/projected/45d5cfa6-bf31-4fad-8003-d28fe9a37183-kube-api-access-kw4f7\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.728278 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.728256 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqg4h\" (UniqueName: \"kubernetes.io/projected/3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8-kube-api-access-mqg4h\") pod \"iptables-alerter-npsp2\" (UID: \"3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8\") " pod="openshift-network-operator/iptables-alerter-npsp2" Nov 01 22:23:52.728368 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.728353 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjzrz\" (UniqueName: \"kubernetes.io/projected/b3a3cc56-6cfb-4092-ba39-5c237ad58c77-kube-api-access-qjzrz\") pod \"ingress-canary-v2l2j\" (UID: \"b3a3cc56-6cfb-4092-ba39-5c237ad58c77\") " pod="openshift-ingress-canary/ingress-canary-v2l2j" Nov 01 22:23:52.728425 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.728412 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxhl8\" (UniqueName: \"kubernetes.io/projected/b439b405-8bca-4ce8-980d-a076cb2f11bf-kube-api-access-lxhl8\") pod \"network-check-target-cfnmp\" (UID: \"b439b405-8bca-4ce8-980d-a076cb2f11bf\") " pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:23:52.728613 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.728595 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkg5t\" (UniqueName: \"kubernetes.io/projected/3af15ebe-c67a-40c5-bac0-24410542413b-kube-api-access-nkg5t\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:23:52.729182 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.729161 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sstcj\" (UniqueName: \"kubernetes.io/projected/179a164c-f6dd-42f9-9428-f241f81afff5-kube-api-access-sstcj\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.729730 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.729711 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ndlv\" (UniqueName: \"kubernetes.io/projected/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-kube-api-access-8ndlv\") pod \"periodic-gathering-sk8f6-sf7bv\" (UID: \"3880904a-615b-44ac-9a2d-7b0f7ad31e0c\") " pod="openshift-insights/periodic-gathering-sk8f6-sf7bv" Nov 01 22:23:52.730351 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.730331 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8wg9\" (UniqueName: \"kubernetes.io/projected/9af8b8a8-47f6-4576-87db-2f0253903e5b-kube-api-access-h8wg9\") pod \"dns-default-srdnl\" (UID: \"9af8b8a8-47f6-4576-87db-2f0253903e5b\") " pod="openshift-dns/dns-default-srdnl" Nov 01 22:23:52.730809 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.730789 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nvbs\" (UniqueName: \"kubernetes.io/projected/cc3de28d-1139-4677-81db-a114c8867933-kube-api-access-6nvbs\") pod \"network-metrics-daemon-xvgrl\" (UID: \"cc3de28d-1139-4677-81db-a114c8867933\") " pod="openshift-multus/network-metrics-daemon-xvgrl" Nov 01 22:23:52.731231 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.731211 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s67jv\" (UniqueName: \"kubernetes.io/projected/5c959caa-7239-4b81-8c19-c867fc6c48e6-kube-api-access-s67jv\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.731571 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.731533 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm2vf\" (UniqueName: \"kubernetes.io/projected/eb7b37b5-9639-45da-b90e-e08dcfac93f3-kube-api-access-vm2vf\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:52.732205 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.732183 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwr44\" (UniqueName: \"kubernetes.io/projected/e296d10d-5064-4f8f-85f0-e6d1b20f1ab8-kube-api-access-gwr44\") pod \"prometheus-operator-9cd89b74c-sxspm\" (UID: \"e296d10d-5064-4f8f-85f0-e6d1b20f1ab8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-sxspm" Nov 01 22:23:52.732537 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.732518 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9q9h\" (UniqueName: \"kubernetes.io/projected/14cdd491-7383-4cc5-ac4f-60a75110d764-kube-api-access-n9q9h\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:23:52.732974 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.732950 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd2dd\" (UniqueName: \"kubernetes.io/projected/682b0c5d-fa6f-4a17-8b72-a1f224cf5fec-kube-api-access-fd2dd\") pod \"node-ca-rkf2j\" (UID: \"682b0c5d-fa6f-4a17-8b72-a1f224cf5fec\") " pod="openshift-image-registry/node-ca-rkf2j" Nov 01 22:23:52.789269 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.789194 5011 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:23:52.789390 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.789282 5011 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:23:52.803715 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.803691 5011 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 22:23:52.803875 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:23:52.803748 5011 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-238.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:23:52.804080 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.804064 5011 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 22:23:52.804118 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:23:52.804105 5011 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.us-east-2.compute.internal" Nov 01 22:23:52.824775 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.824750 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-slash\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.824970 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.824780 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-cni-netd\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.824970 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.824796 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-systemd\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.824970 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.824812 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-sys\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.824970 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.824857 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-run-k8s-cni-cncf-io\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.824970 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.824900 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-slash\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.824970 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.824917 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-sys\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.824970 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.824900 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-systemd\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.824970 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.824900 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-cni-netd\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.824970 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.824904 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-run-k8s-cni-cncf-io\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.824970 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.824957 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-sysctl-conf\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.825385 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.824992 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:23:52.825385 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825014 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-run-ovn\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.825385 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825038 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-cnibin\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.825385 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825076 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:23:52.825385 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825082 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-run-ovn\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.825385 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825094 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-cnibin\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.825385 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825115 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-multus-conf-dir\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.825385 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825135 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-registration-dir\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:23:52.825385 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825151 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fa603801-6c30-4cf1-ace8-30c8723e4a1e-dbus\") pod \"global-pull-secret-syncer-ttjsz\" (UID: \"fa603801-6c30-4cf1-ace8-30c8723e4a1e\") " pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:23:52.825385 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825163 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-sysctl-conf\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.825385 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825190 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-multus-conf-dir\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.825385 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825203 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-registration-dir\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:23:52.825385 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825215 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fa603801-6c30-4cf1-ace8-30c8723e4a1e-dbus\") pod \"global-pull-secret-syncer-ttjsz\" (UID: \"fa603801-6c30-4cf1-ace8-30c8723e4a1e\") " pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:23:52.825385 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825220 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-run-multus-certs\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.825385 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825244 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb7b37b5-9639-45da-b90e-e08dcfac93f3-crio-socket\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:52.825385 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825264 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3af15ebe-c67a-40c5-bac0-24410542413b-system-cni-dir\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:23:52.825385 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825279 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-etc-selinux\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:23:52.825385 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825293 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-host\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.825385 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825275 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-run-multus-certs\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.825385 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825306 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb7b37b5-9639-45da-b90e-e08dcfac93f3-crio-socket\") pod \"insights-runtime-extractor-lcqwg\" (UID: \"eb7b37b5-9639-45da-b90e-e08dcfac93f3\") " pod="openshift-insights/insights-runtime-extractor-lcqwg" Nov 01 22:23:52.825385 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825313 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-socket-dir\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:23:52.825385 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825336 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3af15ebe-c67a-40c5-bac0-24410542413b-system-cni-dir\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:23:52.825385 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825353 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-systemd-units\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.825385 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825360 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-etc-selinux\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:23:52.825385 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825369 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-socket-dir\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:23:52.825385 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825359 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-host\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.825385 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825379 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-cni-bin\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.825385 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825388 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-systemd-units\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.825385 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825401 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-cni-bin\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.825385 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825406 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-var-lib-cni-multus\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825429 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3af15ebe-c67a-40c5-bac0-24410542413b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825453 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-log-socket\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825464 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-var-lib-cni-multus\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825474 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3af15ebe-c67a-40c5-bac0-24410542413b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825475 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fa603801-6c30-4cf1-ace8-30c8723e4a1e-kubelet-config\") pod \"global-pull-secret-syncer-ttjsz\" (UID: \"fa603801-6c30-4cf1-ace8-30c8723e4a1e\") " pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825504 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-run\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825509 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-log-socket\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825506 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fa603801-6c30-4cf1-ace8-30c8723e4a1e-kubelet-config\") pod \"global-pull-secret-syncer-ttjsz\" (UID: \"fa603801-6c30-4cf1-ace8-30c8723e4a1e\") " pod="kube-system/global-pull-secret-syncer-ttjsz" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825527 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-run\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825526 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-var-lib-kubelet\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825556 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-device-dir\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825564 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-var-lib-kubelet\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825577 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-multus-socket-dir-parent\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825585 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-device-dir\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825591 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f71460db-658f-45ed-980f-da65509e2bc5-hosts-file\") pod \"node-resolver-x4k8b\" (UID: \"f71460db-658f-45ed-980f-da65509e2bc5\") " pod="openshift-dns/node-resolver-x4k8b" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825608 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825620 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-multus-socket-dir-parent\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825632 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-hostroot\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825653 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/682b0c5d-fa6f-4a17-8b72-a1f224cf5fec-host\") pod \"node-ca-rkf2j\" (UID: \"682b0c5d-fa6f-4a17-8b72-a1f224cf5fec\") " pod="openshift-image-registry/node-ca-rkf2j" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825674 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-run-netns\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825675 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825615 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f71460db-658f-45ed-980f-da65509e2bc5-hosts-file\") pod \"node-resolver-x4k8b\" (UID: \"f71460db-658f-45ed-980f-da65509e2bc5\") " pod="openshift-dns/node-resolver-x4k8b" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825698 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-hostroot\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825688 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-node-log\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825721 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-node-log\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825731 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-run-netns\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825707 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/682b0c5d-fa6f-4a17-8b72-a1f224cf5fec-host\") pod \"node-ca-rkf2j\" (UID: \"682b0c5d-fa6f-4a17-8b72-a1f224cf5fec\") " pod="openshift-image-registry/node-ca-rkf2j" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825731 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-sysconfig\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825760 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-var-lib-cni-bin\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825790 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-sysconfig\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825809 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-run-openvswitch\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825847 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-sysctl-d\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825864 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-multus-cni-dir\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825885 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3af15ebe-c67a-40c5-bac0-24410542413b-cnibin\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825887 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-run-openvswitch\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825810 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-var-lib-cni-bin\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825911 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-run-ovn-kubernetes\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825927 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-multus-cni-dir\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825932 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3af15ebe-c67a-40c5-bac0-24410542413b-cnibin\") pod \"multus-additional-cni-plugins-jngxk\" (UID: \"3af15ebe-c67a-40c5-bac0-24410542413b\") " pod="openshift-multus/multus-additional-cni-plugins-jngxk" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825938 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-modprobe-d\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825974 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-run-ovn-kubernetes\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825997 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-system-cni-dir\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.825977 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-sysctl-d\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.826042 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-system-cni-dir\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.826060 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-modprobe-d\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.826062 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-var-lib-kubelet\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.826082 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-sys-fs\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.826095 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-kubelet\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.826108 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-run-systemd\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.826125 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-lib-modules\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.826144 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8-host-slash\") pod \"iptables-alerter-npsp2\" (UID: \"3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8\") " pod="openshift-network-operator/iptables-alerter-npsp2" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.826147 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-host-kubelet\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.826150 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/14cdd491-7383-4cc5-ac4f-60a75110d764-sys-fs\") pod \"aws-ebs-csi-driver-node-nmtlr\" (UID: \"14cdd491-7383-4cc5-ac4f-60a75110d764\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmtlr" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.826164 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-run-systemd\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.826169 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-etc-kubernetes\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.826177 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8-host-slash\") pod \"iptables-alerter-npsp2\" (UID: \"3f0ded2e-e6aa-4764-a96b-ff2f41ce54b8\") " pod="openshift-network-operator/iptables-alerter-npsp2" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.826132 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-var-lib-kubelet\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.826201 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-etc-kubernetes\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.826202 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-etc-openvswitch\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.826224 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-etc-openvswitch\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.826224 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-lib-modules\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.826239 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-run-netns\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.826253 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/179a164c-f6dd-42f9-9428-f241f81afff5-host-run-netns\") pod \"multus-84vvx\" (UID: \"179a164c-f6dd-42f9-9428-f241f81afff5\") " pod="openshift-multus/multus-84vvx" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.826257 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-var-lib-openvswitch\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.826277 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45d5cfa6-bf31-4fad-8003-d28fe9a37183-var-lib-openvswitch\") pod \"ovnkube-node-6h899\" (UID: \"45d5cfa6-bf31-4fad-8003-d28fe9a37183\") " pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.826282 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-kubernetes\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.826444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.826328 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c959caa-7239-4b81-8c19-c867fc6c48e6-etc-kubernetes\") pod \"tuned-mhvjj\" (UID: \"5c959caa-7239-4b81-8c19-c867fc6c48e6\") " pod="openshift-cluster-node-tuning-operator/tuned-mhvjj" Nov 01 22:23:52.984561 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:52.984529 5011 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:23:53.794138 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:53.794099 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-sxspm" event={"ID":"e296d10d-5064-4f8f-85f0-e6d1b20f1ab8","Type":"ContainerStarted","Data":"fd53ace507f4c03cb30ae048b923144285c38a9d2c28e7f6fea1c4df56f1e865"} Nov 01 22:23:53.794627 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:53.794174 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-sxspm" event={"ID":"e296d10d-5064-4f8f-85f0-e6d1b20f1ab8","Type":"ContainerStarted","Data":"8df67f671980bd1332c0c44dddfd5147db66ac4b98559785fc4542ee49c69e2c"} Nov 01 22:23:53.796532 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:53.796501 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lcqwg" event={"ID":"eb7b37b5-9639-45da-b90e-e08dcfac93f3","Type":"ContainerStarted","Data":"362e94d5b45321b9526569f6a5a3207afb21f7c91fb0dfb24debc2bafda555f4"} Nov 01 22:23:54.816381 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:54.816330 5011 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tmhm2_3bed4cac-d251-49be-9eb6-ce5cdb6852de/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:54.913039 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:54.913002 5011 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tmhm2_3bed4cac-d251-49be-9eb6-ce5cdb6852de/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:55.007575 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.007528 5011 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tmhm2_3bed4cac-d251-49be-9eb6-ce5cdb6852de/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:55.389912 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.386986 5011 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tmhm2_3bed4cac-d251-49be-9eb6-ce5cdb6852de/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:55.423793 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.423755 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-srdnl" Nov 01 22:23:55.625700 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.625662 5011 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tmhm2_3bed4cac-d251-49be-9eb6-ce5cdb6852de/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:55.682303 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.682272 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4f58c"] Nov 01 22:23:55.685844 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.685811 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4f58c" Nov 01 22:23:55.694375 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.694355 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 22:23:55.694492 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.694465 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 22:23:55.694566 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.694521 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5n9np\"" Nov 01 22:23:55.694566 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.694560 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 22:23:55.746623 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.746593 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca5a7ca1-bb6d-4e12-90da-e0a1831153b0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4f58c\" (UID: \"ca5a7ca1-bb6d-4e12-90da-e0a1831153b0\") " pod="openshift-monitoring/node-exporter-4f58c" Nov 01 22:23:55.746823 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.746658 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca5a7ca1-bb6d-4e12-90da-e0a1831153b0-sys\") pod \"node-exporter-4f58c\" (UID: \"ca5a7ca1-bb6d-4e12-90da-e0a1831153b0\") " pod="openshift-monitoring/node-exporter-4f58c" Nov 01 22:23:55.746823 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.746682 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca5a7ca1-bb6d-4e12-90da-e0a1831153b0-metrics-client-ca\") pod \"node-exporter-4f58c\" (UID: \"ca5a7ca1-bb6d-4e12-90da-e0a1831153b0\") " pod="openshift-monitoring/node-exporter-4f58c" Nov 01 22:23:55.746823 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.746703 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ca5a7ca1-bb6d-4e12-90da-e0a1831153b0-root\") pod \"node-exporter-4f58c\" (UID: \"ca5a7ca1-bb6d-4e12-90da-e0a1831153b0\") " pod="openshift-monitoring/node-exporter-4f58c" Nov 01 22:23:55.746823 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.746726 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ca5a7ca1-bb6d-4e12-90da-e0a1831153b0-node-exporter-tls\") pod \"node-exporter-4f58c\" (UID: \"ca5a7ca1-bb6d-4e12-90da-e0a1831153b0\") " pod="openshift-monitoring/node-exporter-4f58c" Nov 01 22:23:55.746823 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.746746 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ca5a7ca1-bb6d-4e12-90da-e0a1831153b0-node-exporter-accelerators-collector-config\") pod \"node-exporter-4f58c\" (UID: \"ca5a7ca1-bb6d-4e12-90da-e0a1831153b0\") " pod="openshift-monitoring/node-exporter-4f58c" Nov 01 22:23:55.746823 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.746776 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zv7m\" (UniqueName: \"kubernetes.io/projected/ca5a7ca1-bb6d-4e12-90da-e0a1831153b0-kube-api-access-9zv7m\") pod \"node-exporter-4f58c\" (UID: \"ca5a7ca1-bb6d-4e12-90da-e0a1831153b0\") " pod="openshift-monitoring/node-exporter-4f58c" Nov 01 22:23:55.746823 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.746798 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ca5a7ca1-bb6d-4e12-90da-e0a1831153b0-node-exporter-textfile\") pod \"node-exporter-4f58c\" (UID: \"ca5a7ca1-bb6d-4e12-90da-e0a1831153b0\") " pod="openshift-monitoring/node-exporter-4f58c" Nov 01 22:23:55.747130 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.746892 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ca5a7ca1-bb6d-4e12-90da-e0a1831153b0-node-exporter-wtmp\") pod \"node-exporter-4f58c\" (UID: \"ca5a7ca1-bb6d-4e12-90da-e0a1831153b0\") " pod="openshift-monitoring/node-exporter-4f58c" Nov 01 22:23:55.847315 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.847284 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca5a7ca1-bb6d-4e12-90da-e0a1831153b0-sys\") pod \"node-exporter-4f58c\" (UID: \"ca5a7ca1-bb6d-4e12-90da-e0a1831153b0\") " pod="openshift-monitoring/node-exporter-4f58c" Nov 01 22:23:55.847315 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.847318 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca5a7ca1-bb6d-4e12-90da-e0a1831153b0-metrics-client-ca\") pod \"node-exporter-4f58c\" (UID: \"ca5a7ca1-bb6d-4e12-90da-e0a1831153b0\") " pod="openshift-monitoring/node-exporter-4f58c" Nov 01 22:23:55.847817 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.847340 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ca5a7ca1-bb6d-4e12-90da-e0a1831153b0-root\") pod \"node-exporter-4f58c\" (UID: \"ca5a7ca1-bb6d-4e12-90da-e0a1831153b0\") " pod="openshift-monitoring/node-exporter-4f58c" Nov 01 22:23:55.847817 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.847364 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ca5a7ca1-bb6d-4e12-90da-e0a1831153b0-node-exporter-tls\") pod \"node-exporter-4f58c\" (UID: \"ca5a7ca1-bb6d-4e12-90da-e0a1831153b0\") " pod="openshift-monitoring/node-exporter-4f58c" Nov 01 22:23:55.847817 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.847388 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ca5a7ca1-bb6d-4e12-90da-e0a1831153b0-node-exporter-accelerators-collector-config\") pod \"node-exporter-4f58c\" (UID: \"ca5a7ca1-bb6d-4e12-90da-e0a1831153b0\") " pod="openshift-monitoring/node-exporter-4f58c" Nov 01 22:23:55.847817 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.847399 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca5a7ca1-bb6d-4e12-90da-e0a1831153b0-sys\") pod \"node-exporter-4f58c\" (UID: \"ca5a7ca1-bb6d-4e12-90da-e0a1831153b0\") " pod="openshift-monitoring/node-exporter-4f58c" Nov 01 22:23:55.847817 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.847419 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zv7m\" (UniqueName: \"kubernetes.io/projected/ca5a7ca1-bb6d-4e12-90da-e0a1831153b0-kube-api-access-9zv7m\") pod \"node-exporter-4f58c\" (UID: \"ca5a7ca1-bb6d-4e12-90da-e0a1831153b0\") " pod="openshift-monitoring/node-exporter-4f58c" Nov 01 22:23:55.847817 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.847423 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ca5a7ca1-bb6d-4e12-90da-e0a1831153b0-root\") pod \"node-exporter-4f58c\" (UID: \"ca5a7ca1-bb6d-4e12-90da-e0a1831153b0\") " pod="openshift-monitoring/node-exporter-4f58c" Nov 01 22:23:55.847817 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.847446 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ca5a7ca1-bb6d-4e12-90da-e0a1831153b0-node-exporter-textfile\") pod \"node-exporter-4f58c\" (UID: \"ca5a7ca1-bb6d-4e12-90da-e0a1831153b0\") " pod="openshift-monitoring/node-exporter-4f58c" Nov 01 22:23:55.847817 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.847485 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ca5a7ca1-bb6d-4e12-90da-e0a1831153b0-node-exporter-wtmp\") pod \"node-exporter-4f58c\" (UID: \"ca5a7ca1-bb6d-4e12-90da-e0a1831153b0\") " pod="openshift-monitoring/node-exporter-4f58c" Nov 01 22:23:55.847817 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.847522 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca5a7ca1-bb6d-4e12-90da-e0a1831153b0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4f58c\" (UID: \"ca5a7ca1-bb6d-4e12-90da-e0a1831153b0\") " pod="openshift-monitoring/node-exporter-4f58c" Nov 01 22:23:55.847817 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.847704 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ca5a7ca1-bb6d-4e12-90da-e0a1831153b0-node-exporter-wtmp\") pod \"node-exporter-4f58c\" (UID: \"ca5a7ca1-bb6d-4e12-90da-e0a1831153b0\") " pod="openshift-monitoring/node-exporter-4f58c" Nov 01 22:23:55.848247 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.847853 5011 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 22:23:55.848247 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.847941 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ca5a7ca1-bb6d-4e12-90da-e0a1831153b0-node-exporter-textfile\") pod \"node-exporter-4f58c\" (UID: \"ca5a7ca1-bb6d-4e12-90da-e0a1831153b0\") " pod="openshift-monitoring/node-exporter-4f58c" Nov 01 22:23:55.848247 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.847964 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca5a7ca1-bb6d-4e12-90da-e0a1831153b0-metrics-client-ca\") pod \"node-exporter-4f58c\" (UID: \"ca5a7ca1-bb6d-4e12-90da-e0a1831153b0\") " pod="openshift-monitoring/node-exporter-4f58c" Nov 01 22:23:55.848247 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.848040 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ca5a7ca1-bb6d-4e12-90da-e0a1831153b0-node-exporter-accelerators-collector-config\") pod \"node-exporter-4f58c\" (UID: \"ca5a7ca1-bb6d-4e12-90da-e0a1831153b0\") " pod="openshift-monitoring/node-exporter-4f58c" Nov 01 22:23:55.850112 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.850092 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca5a7ca1-bb6d-4e12-90da-e0a1831153b0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4f58c\" (UID: \"ca5a7ca1-bb6d-4e12-90da-e0a1831153b0\") " pod="openshift-monitoring/node-exporter-4f58c" Nov 01 22:23:55.850238 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.850216 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ca5a7ca1-bb6d-4e12-90da-e0a1831153b0-node-exporter-tls\") pod \"node-exporter-4f58c\" (UID: \"ca5a7ca1-bb6d-4e12-90da-e0a1831153b0\") " pod="openshift-monitoring/node-exporter-4f58c" Nov 01 22:23:55.856463 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:55.856435 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zv7m\" (UniqueName: \"kubernetes.io/projected/ca5a7ca1-bb6d-4e12-90da-e0a1831153b0-kube-api-access-9zv7m\") pod \"node-exporter-4f58c\" (UID: \"ca5a7ca1-bb6d-4e12-90da-e0a1831153b0\") " pod="openshift-monitoring/node-exporter-4f58c" Nov 01 22:23:56.001500 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.001421 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4f58c" Nov 01 22:23:56.013433 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:56.013393 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca5a7ca1_bb6d_4e12_90da_e0a1831153b0.slice/crio-845ff9251ba69d946b3213e3e8358b4ae4034d499e48bcb5e77f92bd06bbe60f WatchSource:0}: Error finding container 845ff9251ba69d946b3213e3e8358b4ae4034d499e48bcb5e77f92bd06bbe60f: Status 404 returned error can't find the container with id 845ff9251ba69d946b3213e3e8358b4ae4034d499e48bcb5e77f92bd06bbe60f Nov 01 22:23:56.714049 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.714003 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 22:23:56.717770 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.717747 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.722391 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.721851 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 01 22:23:56.722391 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.722041 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 01 22:23:56.722391 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.722115 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 01 22:23:56.722391 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.722178 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 01 22:23:56.724041 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.723035 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zddmv\"" Nov 01 22:23:56.725060 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.725041 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 01 22:23:56.725166 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.725066 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 01 22:23:56.725166 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.725083 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 01 22:23:56.725166 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.725151 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 01 22:23:56.725279 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.725160 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 01 22:23:56.732092 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.732070 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 22:23:56.754393 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.754363 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5349ab21-2645-4464-a07c-d71e4503808d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.754576 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.754409 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.754576 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.754436 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5349ab21-2645-4464-a07c-d71e4503808d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.754576 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.754556 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5349ab21-2645-4464-a07c-d71e4503808d-config-out\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.754985 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.754599 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5349ab21-2645-4464-a07c-d71e4503808d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.754985 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.754682 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.754985 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.754705 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5349ab21-2645-4464-a07c-d71e4503808d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.754985 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.754741 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-config-volume\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.754985 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.754787 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.754985 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.754808 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-web-config\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.754985 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.754823 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgxh6\" (UniqueName: \"kubernetes.io/projected/5349ab21-2645-4464-a07c-d71e4503808d-kube-api-access-mgxh6\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.754985 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.754871 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.754985 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.754889 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.807561 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.807527 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4f58c" event={"ID":"ca5a7ca1-bb6d-4e12-90da-e0a1831153b0","Type":"ContainerStarted","Data":"845ff9251ba69d946b3213e3e8358b4ae4034d499e48bcb5e77f92bd06bbe60f"} Nov 01 22:23:56.856168 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.856137 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-config-volume\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.856590 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.856198 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.856590 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.856221 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-web-config\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.856590 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.856242 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgxh6\" (UniqueName: \"kubernetes.io/projected/5349ab21-2645-4464-a07c-d71e4503808d-kube-api-access-mgxh6\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.856590 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.856274 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.856590 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.856298 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.856590 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.856323 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5349ab21-2645-4464-a07c-d71e4503808d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.856590 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.856350 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.856590 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.856370 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5349ab21-2645-4464-a07c-d71e4503808d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.856590 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.856399 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5349ab21-2645-4464-a07c-d71e4503808d-config-out\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.856590 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.856419 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5349ab21-2645-4464-a07c-d71e4503808d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.856590 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.856456 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.856590 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.856481 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5349ab21-2645-4464-a07c-d71e4503808d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.857169 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:23:56.857098 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5349ab21-2645-4464-a07c-d71e4503808d-alertmanager-trusted-ca-bundle podName:5349ab21-2645-4464-a07c-d71e4503808d nodeName:}" failed. No retries permitted until 2026-11-01 22:23:57.357066822 +0000 UTC m=+5.761657451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/5349ab21-2645-4464-a07c-d71e4503808d-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "5349ab21-2645-4464-a07c-d71e4503808d") : configmap references non-existent config key: ca-bundle.crt Nov 01 22:23:56.858309 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.858047 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5349ab21-2645-4464-a07c-d71e4503808d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.858605 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.858580 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5349ab21-2645-4464-a07c-d71e4503808d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.861541 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.861516 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-config-volume\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.863210 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.863182 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.863535 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.863486 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5349ab21-2645-4464-a07c-d71e4503808d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.864049 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.864028 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.864225 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.864203 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.864492 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.864465 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-web-config\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.865529 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.865508 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.867971 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.867950 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5349ab21-2645-4464-a07c-d71e4503808d-config-out\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.868399 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.868376 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgxh6\" (UniqueName: \"kubernetes.io/projected/5349ab21-2645-4464-a07c-d71e4503808d-kube-api-access-mgxh6\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:56.868595 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:56.868574 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:57.089411 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:57.089317 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-hpj5f-7thv7"] Nov 01 22:23:57.092938 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:57.092910 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hpj5f-7thv7" Nov 01 22:23:57.101856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:57.101489 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-hpj5f-7thv7"] Nov 01 22:23:57.159508 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:57.159463 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxs2f\" (UniqueName: \"kubernetes.io/projected/ed5e5b02-53ae-4da1-9ed5-43815cb00ce0-kube-api-access-pxs2f\") pod \"periodic-gathering-hpj5f-7thv7\" (UID: \"ed5e5b02-53ae-4da1-9ed5-43815cb00ce0\") " pod="openshift-insights/periodic-gathering-hpj5f-7thv7" Nov 01 22:23:57.159508 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:57.159523 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ed5e5b02-53ae-4da1-9ed5-43815cb00ce0-archives-path\") pod \"periodic-gathering-hpj5f-7thv7\" (UID: \"ed5e5b02-53ae-4da1-9ed5-43815cb00ce0\") " pod="openshift-insights/periodic-gathering-hpj5f-7thv7" Nov 01 22:23:57.159744 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:57.159726 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed5e5b02-53ae-4da1-9ed5-43815cb00ce0-service-ca-bundle\") pod \"periodic-gathering-hpj5f-7thv7\" (UID: \"ed5e5b02-53ae-4da1-9ed5-43815cb00ce0\") " pod="openshift-insights/periodic-gathering-hpj5f-7thv7" Nov 01 22:23:57.159794 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:57.159778 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed5e5b02-53ae-4da1-9ed5-43815cb00ce0-trusted-ca-bundle\") pod \"periodic-gathering-hpj5f-7thv7\" (UID: \"ed5e5b02-53ae-4da1-9ed5-43815cb00ce0\") " pod="openshift-insights/periodic-gathering-hpj5f-7thv7" Nov 01 22:23:57.260609 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:57.260580 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed5e5b02-53ae-4da1-9ed5-43815cb00ce0-trusted-ca-bundle\") pod \"periodic-gathering-hpj5f-7thv7\" (UID: \"ed5e5b02-53ae-4da1-9ed5-43815cb00ce0\") " pod="openshift-insights/periodic-gathering-hpj5f-7thv7" Nov 01 22:23:57.260727 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:57.260635 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxs2f\" (UniqueName: \"kubernetes.io/projected/ed5e5b02-53ae-4da1-9ed5-43815cb00ce0-kube-api-access-pxs2f\") pod \"periodic-gathering-hpj5f-7thv7\" (UID: \"ed5e5b02-53ae-4da1-9ed5-43815cb00ce0\") " pod="openshift-insights/periodic-gathering-hpj5f-7thv7" Nov 01 22:23:57.260727 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:57.260672 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ed5e5b02-53ae-4da1-9ed5-43815cb00ce0-archives-path\") pod \"periodic-gathering-hpj5f-7thv7\" (UID: \"ed5e5b02-53ae-4da1-9ed5-43815cb00ce0\") " pod="openshift-insights/periodic-gathering-hpj5f-7thv7" Nov 01 22:23:57.260864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:57.260740 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed5e5b02-53ae-4da1-9ed5-43815cb00ce0-service-ca-bundle\") pod \"periodic-gathering-hpj5f-7thv7\" (UID: \"ed5e5b02-53ae-4da1-9ed5-43815cb00ce0\") " pod="openshift-insights/periodic-gathering-hpj5f-7thv7" Nov 01 22:23:57.261063 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:57.261043 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ed5e5b02-53ae-4da1-9ed5-43815cb00ce0-archives-path\") pod \"periodic-gathering-hpj5f-7thv7\" (UID: \"ed5e5b02-53ae-4da1-9ed5-43815cb00ce0\") " pod="openshift-insights/periodic-gathering-hpj5f-7thv7" Nov 01 22:23:57.261317 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:57.261292 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed5e5b02-53ae-4da1-9ed5-43815cb00ce0-trusted-ca-bundle\") pod \"periodic-gathering-hpj5f-7thv7\" (UID: \"ed5e5b02-53ae-4da1-9ed5-43815cb00ce0\") " pod="openshift-insights/periodic-gathering-hpj5f-7thv7" Nov 01 22:23:57.261317 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:57.261310 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed5e5b02-53ae-4da1-9ed5-43815cb00ce0-service-ca-bundle\") pod \"periodic-gathering-hpj5f-7thv7\" (UID: \"ed5e5b02-53ae-4da1-9ed5-43815cb00ce0\") " pod="openshift-insights/periodic-gathering-hpj5f-7thv7" Nov 01 22:23:57.270221 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:57.270196 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxs2f\" (UniqueName: \"kubernetes.io/projected/ed5e5b02-53ae-4da1-9ed5-43815cb00ce0-kube-api-access-pxs2f\") pod \"periodic-gathering-hpj5f-7thv7\" (UID: \"ed5e5b02-53ae-4da1-9ed5-43815cb00ce0\") " pod="openshift-insights/periodic-gathering-hpj5f-7thv7" Nov 01 22:23:57.307281 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:57.307255 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:23:57.361451 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:57.361362 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5349ab21-2645-4464-a07c-d71e4503808d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:57.362433 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:57.362406 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5349ab21-2645-4464-a07c-d71e4503808d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:57.407343 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:57.407305 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hpj5f-7thv7" Nov 01 22:23:57.647868 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:57.647778 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:23:57.685380 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:57.685352 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-hpj5f-7thv7"] Nov 01 22:23:57.688588 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:57.688562 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded5e5b02_53ae_4da1_9ed5_43815cb00ce0.slice/crio-d91f5adb2d727f28228ccaf880a918818cd5d1ef1a3bb199834d6806e66a351f WatchSource:0}: Error finding container d91f5adb2d727f28228ccaf880a918818cd5d1ef1a3bb199834d6806e66a351f: Status 404 returned error can't find the container with id d91f5adb2d727f28228ccaf880a918818cd5d1ef1a3bb199834d6806e66a351f Nov 01 22:23:57.797674 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:57.797636 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:23:57.817042 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:57.817013 5011 generic.go:356] "Generic (PLEG): container finished" podID="ca5a7ca1-bb6d-4e12-90da-e0a1831153b0" containerID="db8f9fee56499cb0c43b85addb570edc003d0637e99395e5c12c297cdcb52ef1" exitCode=0 Nov 01 22:23:57.817388 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:57.817099 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4f58c" event={"ID":"ca5a7ca1-bb6d-4e12-90da-e0a1831153b0","Type":"ContainerDied","Data":"db8f9fee56499cb0c43b85addb570edc003d0637e99395e5c12c297cdcb52ef1"} Nov 01 22:23:57.819303 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:57.819279 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hpj5f-7thv7" event={"ID":"ed5e5b02-53ae-4da1-9ed5-43815cb00ce0","Type":"ContainerStarted","Data":"6ffa13f23e352c981e985300b4317eafa0a656f0533206363a7e3d3a48d9c8e0"} Nov 01 22:23:57.819418 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:57.819322 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hpj5f-7thv7" event={"ID":"ed5e5b02-53ae-4da1-9ed5-43815cb00ce0","Type":"ContainerStarted","Data":"d91f5adb2d727f28228ccaf880a918818cd5d1ef1a3bb199834d6806e66a351f"} Nov 01 22:23:57.852137 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:57.852065 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-hpj5f-7thv7" podStartSLOduration=0.852046507 podStartE2EDuration="852.046507ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:57.851284767 +0000 UTC m=+6.255875414" watchObservedRunningTime="2026-11-01 22:23:57.852046507 +0000 UTC m=+6.256637140" Nov 01 22:23:57.952409 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:57.952371 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 22:23:57.955797 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:23:57.955732 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5349ab21_2645_4464_a07c_d71e4503808d.slice/crio-d36d7430cc118bd77f441e3cd452ae5c6c30a61deef5c59bf02d915b7855e7c9 WatchSource:0}: Error finding container d36d7430cc118bd77f441e3cd452ae5c6c30a61deef5c59bf02d915b7855e7c9: Status 404 returned error can't find the container with id d36d7430cc118bd77f441e3cd452ae5c6c30a61deef5c59bf02d915b7855e7c9 Nov 01 22:23:58.426486 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:58.426453 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-srdnl" Nov 01 22:23:58.824478 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:58.824394 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5349ab21-2645-4464-a07c-d71e4503808d","Type":"ContainerStarted","Data":"d36d7430cc118bd77f441e3cd452ae5c6c30a61deef5c59bf02d915b7855e7c9"} Nov 01 22:23:58.827799 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:58.827766 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4f58c" event={"ID":"ca5a7ca1-bb6d-4e12-90da-e0a1831153b0","Type":"ContainerStarted","Data":"c514968084b7af240e427181c287cdbbd0033140fcee91f72273014485bcfe2d"} Nov 01 22:23:58.827982 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:58.827825 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4f58c" event={"ID":"ca5a7ca1-bb6d-4e12-90da-e0a1831153b0","Type":"ContainerStarted","Data":"d8d470ccbb53568a7e1f8aacce74c038f48276ea2d944d87b6975293c21f94e2"} Nov 01 22:23:59.832618 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:59.832581 5011 generic.go:356] "Generic (PLEG): container finished" podID="5349ab21-2645-4464-a07c-d71e4503808d" containerID="d331363f4284d8fd77ab657eafb70be3c11dfa2ad3b3e96a48a46020fbc2b126" exitCode=0 Nov 01 22:23:59.833105 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:59.832676 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5349ab21-2645-4464-a07c-d71e4503808d","Type":"ContainerDied","Data":"d331363f4284d8fd77ab657eafb70be3c11dfa2ad3b3e96a48a46020fbc2b126"} Nov 01 22:23:59.857318 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:23:59.857264 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4f58c" podStartSLOduration=3.564974034 podStartE2EDuration="4.857249227s" totalImagesPullingTime="1.219786958s" totalInitContainerRuntime="72.488235ms" isStatefulPod=true podCreationTimestamp="2026-11-01 22:23:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:58.856168924 +0000 UTC m=+7.260759548" watchObservedRunningTime="2026-11-01 22:23:59.857249227 +0000 UTC m=+8.261839860" Nov 01 22:24:00.056096 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.056058 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq"] Nov 01 22:24:00.059267 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.059250 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq" Nov 01 22:24:00.063473 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.063451 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 01 22:24:00.063591 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.063493 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 01 22:24:00.068911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.068888 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq"] Nov 01 22:24:00.105566 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.105476 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 01 22:24:00.105566 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.105508 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4fkj446agm1v0\"" Nov 01 22:24:00.105566 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.105510 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 01 22:24:00.105566 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.105554 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-bjcvj\"" Nov 01 22:24:00.184243 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.184198 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ac8d3bc4-33c4-4843-a23b-91c6908a45d1-audit-log\") pod \"metrics-server-66f8d6ccdb-f9xnq\" (UID: \"ac8d3bc4-33c4-4843-a23b-91c6908a45d1\") " pod="openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq" Nov 01 22:24:00.184425 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.184262 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac8d3bc4-33c4-4843-a23b-91c6908a45d1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-66f8d6ccdb-f9xnq\" (UID: \"ac8d3bc4-33c4-4843-a23b-91c6908a45d1\") " pod="openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq" Nov 01 22:24:00.184425 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.184338 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ac8d3bc4-33c4-4843-a23b-91c6908a45d1-secret-metrics-server-client-certs\") pod \"metrics-server-66f8d6ccdb-f9xnq\" (UID: \"ac8d3bc4-33c4-4843-a23b-91c6908a45d1\") " pod="openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq" Nov 01 22:24:00.184425 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.184368 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8h9r\" (UniqueName: \"kubernetes.io/projected/ac8d3bc4-33c4-4843-a23b-91c6908a45d1-kube-api-access-f8h9r\") pod \"metrics-server-66f8d6ccdb-f9xnq\" (UID: \"ac8d3bc4-33c4-4843-a23b-91c6908a45d1\") " pod="openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq" Nov 01 22:24:00.184425 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.184392 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ac8d3bc4-33c4-4843-a23b-91c6908a45d1-secret-metrics-server-tls\") pod \"metrics-server-66f8d6ccdb-f9xnq\" (UID: \"ac8d3bc4-33c4-4843-a23b-91c6908a45d1\") " pod="openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq" Nov 01 22:24:00.184579 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.184479 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ac8d3bc4-33c4-4843-a23b-91c6908a45d1-metrics-server-audit-profiles\") pod \"metrics-server-66f8d6ccdb-f9xnq\" (UID: \"ac8d3bc4-33c4-4843-a23b-91c6908a45d1\") " pod="openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq" Nov 01 22:24:00.184579 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.184546 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ac8d3bc4-33c4-4843-a23b-91c6908a45d1-client-ca-bundle\") pod \"metrics-server-66f8d6ccdb-f9xnq\" (UID: \"ac8d3bc4-33c4-4843-a23b-91c6908a45d1\") " pod="openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq" Nov 01 22:24:00.285512 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.285470 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ac8d3bc4-33c4-4843-a23b-91c6908a45d1-metrics-server-audit-profiles\") pod \"metrics-server-66f8d6ccdb-f9xnq\" (UID: \"ac8d3bc4-33c4-4843-a23b-91c6908a45d1\") " pod="openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq" Nov 01 22:24:00.285681 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.285520 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ac8d3bc4-33c4-4843-a23b-91c6908a45d1-client-ca-bundle\") pod \"metrics-server-66f8d6ccdb-f9xnq\" (UID: \"ac8d3bc4-33c4-4843-a23b-91c6908a45d1\") " pod="openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq" Nov 01 22:24:00.285681 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.285650 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ac8d3bc4-33c4-4843-a23b-91c6908a45d1-audit-log\") pod \"metrics-server-66f8d6ccdb-f9xnq\" (UID: \"ac8d3bc4-33c4-4843-a23b-91c6908a45d1\") " pod="openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq" Nov 01 22:24:00.285811 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.285718 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac8d3bc4-33c4-4843-a23b-91c6908a45d1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-66f8d6ccdb-f9xnq\" (UID: \"ac8d3bc4-33c4-4843-a23b-91c6908a45d1\") " pod="openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq" Nov 01 22:24:00.285811 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.285764 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ac8d3bc4-33c4-4843-a23b-91c6908a45d1-secret-metrics-server-client-certs\") pod \"metrics-server-66f8d6ccdb-f9xnq\" (UID: \"ac8d3bc4-33c4-4843-a23b-91c6908a45d1\") " pod="openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq" Nov 01 22:24:00.285811 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.285793 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8h9r\" (UniqueName: \"kubernetes.io/projected/ac8d3bc4-33c4-4843-a23b-91c6908a45d1-kube-api-access-f8h9r\") pod \"metrics-server-66f8d6ccdb-f9xnq\" (UID: \"ac8d3bc4-33c4-4843-a23b-91c6908a45d1\") " pod="openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq" Nov 01 22:24:00.285986 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.285823 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ac8d3bc4-33c4-4843-a23b-91c6908a45d1-secret-metrics-server-tls\") pod \"metrics-server-66f8d6ccdb-f9xnq\" (UID: \"ac8d3bc4-33c4-4843-a23b-91c6908a45d1\") " pod="openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq" Nov 01 22:24:00.286118 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.286094 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ac8d3bc4-33c4-4843-a23b-91c6908a45d1-audit-log\") pod \"metrics-server-66f8d6ccdb-f9xnq\" (UID: \"ac8d3bc4-33c4-4843-a23b-91c6908a45d1\") " pod="openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq" Nov 01 22:24:00.286611 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.286584 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ac8d3bc4-33c4-4843-a23b-91c6908a45d1-metrics-server-audit-profiles\") pod \"metrics-server-66f8d6ccdb-f9xnq\" (UID: \"ac8d3bc4-33c4-4843-a23b-91c6908a45d1\") " pod="openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq" Nov 01 22:24:00.286706 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.286635 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac8d3bc4-33c4-4843-a23b-91c6908a45d1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-66f8d6ccdb-f9xnq\" (UID: \"ac8d3bc4-33c4-4843-a23b-91c6908a45d1\") " pod="openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq" Nov 01 22:24:00.288201 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.288178 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ac8d3bc4-33c4-4843-a23b-91c6908a45d1-secret-metrics-server-client-certs\") pod \"metrics-server-66f8d6ccdb-f9xnq\" (UID: \"ac8d3bc4-33c4-4843-a23b-91c6908a45d1\") " pod="openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq" Nov 01 22:24:00.288282 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.288260 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ac8d3bc4-33c4-4843-a23b-91c6908a45d1-client-ca-bundle\") pod \"metrics-server-66f8d6ccdb-f9xnq\" (UID: \"ac8d3bc4-33c4-4843-a23b-91c6908a45d1\") " pod="openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq" Nov 01 22:24:00.288317 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.288291 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ac8d3bc4-33c4-4843-a23b-91c6908a45d1-secret-metrics-server-tls\") pod \"metrics-server-66f8d6ccdb-f9xnq\" (UID: \"ac8d3bc4-33c4-4843-a23b-91c6908a45d1\") " pod="openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq" Nov 01 22:24:00.293270 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.293249 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8h9r\" (UniqueName: \"kubernetes.io/projected/ac8d3bc4-33c4-4843-a23b-91c6908a45d1-kube-api-access-f8h9r\") pod \"metrics-server-66f8d6ccdb-f9xnq\" (UID: \"ac8d3bc4-33c4-4843-a23b-91c6908a45d1\") " pod="openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq" Nov 01 22:24:00.370461 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.370381 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq" Nov 01 22:24:00.397392 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.397353 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-h4shp"] Nov 01 22:24:00.400947 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.400923 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-h4shp" Nov 01 22:24:00.409175 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.409152 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 01 22:24:00.409490 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.409464 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-8t22g\"" Nov 01 22:24:00.410241 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.410160 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-h4shp"] Nov 01 22:24:00.487767 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.487734 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2rn7\" (UniqueName: \"kubernetes.io/projected/d9be2a9d-8cd7-4fc4-ab6a-8b3c7793c5bd-kube-api-access-c2rn7\") pod \"monitoring-plugin-77f4f76594-h4shp\" (UID: \"d9be2a9d-8cd7-4fc4-ab6a-8b3c7793c5bd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-h4shp" Nov 01 22:24:00.487927 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.487781 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d9be2a9d-8cd7-4fc4-ab6a-8b3c7793c5bd-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-h4shp\" (UID: \"d9be2a9d-8cd7-4fc4-ab6a-8b3c7793c5bd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-h4shp" Nov 01 22:24:00.578280 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.578250 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:24:00.578474 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.578461 5011 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 22:24:00.588123 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.588094 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2rn7\" (UniqueName: \"kubernetes.io/projected/d9be2a9d-8cd7-4fc4-ab6a-8b3c7793c5bd-kube-api-access-c2rn7\") pod \"monitoring-plugin-77f4f76594-h4shp\" (UID: \"d9be2a9d-8cd7-4fc4-ab6a-8b3c7793c5bd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-h4shp" Nov 01 22:24:00.588271 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.588134 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d9be2a9d-8cd7-4fc4-ab6a-8b3c7793c5bd-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-h4shp\" (UID: \"d9be2a9d-8cd7-4fc4-ab6a-8b3c7793c5bd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-h4shp" Nov 01 22:24:00.590772 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.590740 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d9be2a9d-8cd7-4fc4-ab6a-8b3c7793c5bd-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-h4shp\" (UID: \"d9be2a9d-8cd7-4fc4-ab6a-8b3c7793c5bd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-h4shp" Nov 01 22:24:00.596429 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.596405 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2rn7\" (UniqueName: \"kubernetes.io/projected/d9be2a9d-8cd7-4fc4-ab6a-8b3c7793c5bd-kube-api-access-c2rn7\") pod \"monitoring-plugin-77f4f76594-h4shp\" (UID: \"d9be2a9d-8cd7-4fc4-ab6a-8b3c7793c5bd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-h4shp" Nov 01 22:24:00.650396 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.650316 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq"] Nov 01 22:24:00.653370 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:24:00.653344 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac8d3bc4_33c4_4843_a23b_91c6908a45d1.slice/crio-931af1c8952e501b95953f977d7d1369df177963b0bd2c56bdcf5b48b0be746c WatchSource:0}: Error finding container 931af1c8952e501b95953f977d7d1369df177963b0bd2c56bdcf5b48b0be746c: Status 404 returned error can't find the container with id 931af1c8952e501b95953f977d7d1369df177963b0bd2c56bdcf5b48b0be746c Nov 01 22:24:00.714769 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.714736 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-h4shp" Nov 01 22:24:00.837516 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.837481 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq" event={"ID":"ac8d3bc4-33c4-4843-a23b-91c6908a45d1","Type":"ContainerStarted","Data":"931af1c8952e501b95953f977d7d1369df177963b0bd2c56bdcf5b48b0be746c"} Nov 01 22:24:00.987045 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:00.987015 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-h4shp"] Nov 01 22:24:00.989728 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:24:00.989705 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9be2a9d_8cd7_4fc4_ab6a_8b3c7793c5bd.slice/crio-21f105375dc25511ce5b4fc722c4759d6dcfc5f891ea6739040f21697d26ad27 WatchSource:0}: Error finding container 21f105375dc25511ce5b4fc722c4759d6dcfc5f891ea6739040f21697d26ad27: Status 404 returned error can't find the container with id 21f105375dc25511ce5b4fc722c4759d6dcfc5f891ea6739040f21697d26ad27 Nov 01 22:24:01.401335 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:01.401222 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-srdnl_9af8b8a8-47f6-4576-87db-2f0253903e5b/dns/0.log" Nov 01 22:24:01.601316 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:01.601290 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-srdnl_9af8b8a8-47f6-4576-87db-2f0253903e5b/kube-rbac-proxy/0.log" Nov 01 22:24:01.853764 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:01.853651 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-h4shp" event={"ID":"d9be2a9d-8cd7-4fc4-ab6a-8b3c7793c5bd","Type":"ContainerStarted","Data":"21f105375dc25511ce5b4fc722c4759d6dcfc5f891ea6739040f21697d26ad27"} Nov 01 22:24:02.201719 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:02.201687 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x4k8b_f71460db-658f-45ed-980f-da65509e2bc5/dns-node-resolver/0.log" Nov 01 22:24:02.801965 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:02.801921 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v2l2j_b3a3cc56-6cfb-4092-ba39-5c237ad58c77/serve-healthcheck-canary/0.log" Nov 01 22:24:02.858312 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:02.858269 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq" event={"ID":"ac8d3bc4-33c4-4843-a23b-91c6908a45d1","Type":"ContainerStarted","Data":"c705d5918e0fbec3b3ebcc8d20640d58997a6de93d279f1c378fb193a280edf7"} Nov 01 22:24:02.875198 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:02.875139 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq" podStartSLOduration=1.356784654 podStartE2EDuration="2.875123786s" totalImagesPullingTime="1.518339132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:02.874273811 +0000 UTC m=+11.278864472" watchObservedRunningTime="2026-11-01 22:24:02.875123786 +0000 UTC m=+11.279714418" Nov 01 22:24:03.868146 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:03.868109 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5349ab21-2645-4464-a07c-d71e4503808d","Type":"ContainerStarted","Data":"93e0505dc7ebcad2d5e61ba5ed9d5c84fd4a491847687ecd02182316d836ecba"} Nov 01 22:24:03.868565 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:03.868536 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5349ab21-2645-4464-a07c-d71e4503808d","Type":"ContainerStarted","Data":"6bc3a0740c090bbc8a3983f168e06b3cf7a0696dd1f9c76aae0c662e74f282c5"} Nov 01 22:24:03.876036 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:03.876005 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5349ab21-2645-4464-a07c-d71e4503808d","Type":"ContainerStarted","Data":"8873b887bb93503b0e9ac0ca4e5d30e30ef27bfc0a1844a03d250cfbdba3f5e9"} Nov 01 22:24:03.876170 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:03.876079 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5349ab21-2645-4464-a07c-d71e4503808d","Type":"ContainerStarted","Data":"acac8477146c8e4ec5bddf7d1684ff493357d1a02d26748cd5540211f41a5a51"} Nov 01 22:24:03.876170 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:03.876133 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5349ab21-2645-4464-a07c-d71e4503808d","Type":"ContainerStarted","Data":"a7223bd264b3d6e9b9360c895ce6d4adb5d4d9ea532ae55d7892b6c2896c00d2"} Nov 01 22:24:04.537408 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.537370 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-777f7cff89-wl4bq"] Nov 01 22:24:04.540428 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.540412 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-777f7cff89-wl4bq" Nov 01 22:24:04.543982 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.543961 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 01 22:24:04.544601 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.544581 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 01 22:24:04.544722 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.544628 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 01 22:24:04.544822 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.544808 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 01 22:24:04.544909 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.544822 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-j4zfh\"" Nov 01 22:24:04.544942 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.544923 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 01 22:24:04.545167 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.545155 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 01 22:24:04.545882 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.545850 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 01 22:24:04.548474 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.548453 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 01 22:24:04.551986 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.551969 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-777f7cff89-wl4bq"] Nov 01 22:24:04.622987 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.622951 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfc81400-7732-4692-84b1-9b9cd92e13ea-trusted-ca-bundle\") pod \"console-777f7cff89-wl4bq\" (UID: \"cfc81400-7732-4692-84b1-9b9cd92e13ea\") " pod="openshift-console/console-777f7cff89-wl4bq" Nov 01 22:24:04.623165 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.622998 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cfc81400-7732-4692-84b1-9b9cd92e13ea-console-serving-cert\") pod \"console-777f7cff89-wl4bq\" (UID: \"cfc81400-7732-4692-84b1-9b9cd92e13ea\") " pod="openshift-console/console-777f7cff89-wl4bq" Nov 01 22:24:04.623165 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.623015 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cfc81400-7732-4692-84b1-9b9cd92e13ea-oauth-serving-cert\") pod \"console-777f7cff89-wl4bq\" (UID: \"cfc81400-7732-4692-84b1-9b9cd92e13ea\") " pod="openshift-console/console-777f7cff89-wl4bq" Nov 01 22:24:04.623165 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.623072 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cfc81400-7732-4692-84b1-9b9cd92e13ea-console-config\") pod \"console-777f7cff89-wl4bq\" (UID: \"cfc81400-7732-4692-84b1-9b9cd92e13ea\") " pod="openshift-console/console-777f7cff89-wl4bq" Nov 01 22:24:04.623267 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.623165 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22blp\" (UniqueName: \"kubernetes.io/projected/cfc81400-7732-4692-84b1-9b9cd92e13ea-kube-api-access-22blp\") pod \"console-777f7cff89-wl4bq\" (UID: \"cfc81400-7732-4692-84b1-9b9cd92e13ea\") " pod="openshift-console/console-777f7cff89-wl4bq" Nov 01 22:24:04.623267 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.623200 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfc81400-7732-4692-84b1-9b9cd92e13ea-tmp\") pod \"console-777f7cff89-wl4bq\" (UID: \"cfc81400-7732-4692-84b1-9b9cd92e13ea\") " pod="openshift-console/console-777f7cff89-wl4bq" Nov 01 22:24:04.623322 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.623287 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cfc81400-7732-4692-84b1-9b9cd92e13ea-console-oauth-config\") pod \"console-777f7cff89-wl4bq\" (UID: \"cfc81400-7732-4692-84b1-9b9cd92e13ea\") " pod="openshift-console/console-777f7cff89-wl4bq" Nov 01 22:24:04.623322 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.623312 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cfc81400-7732-4692-84b1-9b9cd92e13ea-service-ca\") pod \"console-777f7cff89-wl4bq\" (UID: \"cfc81400-7732-4692-84b1-9b9cd92e13ea\") " pod="openshift-console/console-777f7cff89-wl4bq" Nov 01 22:24:04.723681 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.723639 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cfc81400-7732-4692-84b1-9b9cd92e13ea-service-ca\") pod \"console-777f7cff89-wl4bq\" (UID: \"cfc81400-7732-4692-84b1-9b9cd92e13ea\") " pod="openshift-console/console-777f7cff89-wl4bq" Nov 01 22:24:04.723681 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.723682 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfc81400-7732-4692-84b1-9b9cd92e13ea-trusted-ca-bundle\") pod \"console-777f7cff89-wl4bq\" (UID: \"cfc81400-7732-4692-84b1-9b9cd92e13ea\") " pod="openshift-console/console-777f7cff89-wl4bq" Nov 01 22:24:04.723959 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.723705 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cfc81400-7732-4692-84b1-9b9cd92e13ea-console-serving-cert\") pod \"console-777f7cff89-wl4bq\" (UID: \"cfc81400-7732-4692-84b1-9b9cd92e13ea\") " pod="openshift-console/console-777f7cff89-wl4bq" Nov 01 22:24:04.723959 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.723729 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cfc81400-7732-4692-84b1-9b9cd92e13ea-oauth-serving-cert\") pod \"console-777f7cff89-wl4bq\" (UID: \"cfc81400-7732-4692-84b1-9b9cd92e13ea\") " pod="openshift-console/console-777f7cff89-wl4bq" Nov 01 22:24:04.723959 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.723762 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cfc81400-7732-4692-84b1-9b9cd92e13ea-console-config\") pod \"console-777f7cff89-wl4bq\" (UID: \"cfc81400-7732-4692-84b1-9b9cd92e13ea\") " pod="openshift-console/console-777f7cff89-wl4bq" Nov 01 22:24:04.723959 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.723806 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22blp\" (UniqueName: \"kubernetes.io/projected/cfc81400-7732-4692-84b1-9b9cd92e13ea-kube-api-access-22blp\") pod \"console-777f7cff89-wl4bq\" (UID: \"cfc81400-7732-4692-84b1-9b9cd92e13ea\") " pod="openshift-console/console-777f7cff89-wl4bq" Nov 01 22:24:04.723959 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.723870 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfc81400-7732-4692-84b1-9b9cd92e13ea-tmp\") pod \"console-777f7cff89-wl4bq\" (UID: \"cfc81400-7732-4692-84b1-9b9cd92e13ea\") " pod="openshift-console/console-777f7cff89-wl4bq" Nov 01 22:24:04.723959 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.723920 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cfc81400-7732-4692-84b1-9b9cd92e13ea-console-oauth-config\") pod \"console-777f7cff89-wl4bq\" (UID: \"cfc81400-7732-4692-84b1-9b9cd92e13ea\") " pod="openshift-console/console-777f7cff89-wl4bq" Nov 01 22:24:04.724376 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.724332 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfc81400-7732-4692-84b1-9b9cd92e13ea-tmp\") pod \"console-777f7cff89-wl4bq\" (UID: \"cfc81400-7732-4692-84b1-9b9cd92e13ea\") " pod="openshift-console/console-777f7cff89-wl4bq" Nov 01 22:24:04.724529 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.724510 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cfc81400-7732-4692-84b1-9b9cd92e13ea-service-ca\") pod \"console-777f7cff89-wl4bq\" (UID: \"cfc81400-7732-4692-84b1-9b9cd92e13ea\") " pod="openshift-console/console-777f7cff89-wl4bq" Nov 01 22:24:04.724650 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.724630 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfc81400-7732-4692-84b1-9b9cd92e13ea-trusted-ca-bundle\") pod \"console-777f7cff89-wl4bq\" (UID: \"cfc81400-7732-4692-84b1-9b9cd92e13ea\") " pod="openshift-console/console-777f7cff89-wl4bq" Nov 01 22:24:04.724719 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.724608 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cfc81400-7732-4692-84b1-9b9cd92e13ea-oauth-serving-cert\") pod \"console-777f7cff89-wl4bq\" (UID: \"cfc81400-7732-4692-84b1-9b9cd92e13ea\") " pod="openshift-console/console-777f7cff89-wl4bq" Nov 01 22:24:04.724977 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.724952 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cfc81400-7732-4692-84b1-9b9cd92e13ea-console-config\") pod \"console-777f7cff89-wl4bq\" (UID: \"cfc81400-7732-4692-84b1-9b9cd92e13ea\") " pod="openshift-console/console-777f7cff89-wl4bq" Nov 01 22:24:04.726436 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.726414 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cfc81400-7732-4692-84b1-9b9cd92e13ea-console-serving-cert\") pod \"console-777f7cff89-wl4bq\" (UID: \"cfc81400-7732-4692-84b1-9b9cd92e13ea\") " pod="openshift-console/console-777f7cff89-wl4bq" Nov 01 22:24:04.726436 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.726432 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cfc81400-7732-4692-84b1-9b9cd92e13ea-console-oauth-config\") pod \"console-777f7cff89-wl4bq\" (UID: \"cfc81400-7732-4692-84b1-9b9cd92e13ea\") " pod="openshift-console/console-777f7cff89-wl4bq" Nov 01 22:24:04.731476 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.731458 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22blp\" (UniqueName: \"kubernetes.io/projected/cfc81400-7732-4692-84b1-9b9cd92e13ea-kube-api-access-22blp\") pod \"console-777f7cff89-wl4bq\" (UID: \"cfc81400-7732-4692-84b1-9b9cd92e13ea\") " pod="openshift-console/console-777f7cff89-wl4bq" Nov 01 22:24:04.852527 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.852447 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-777f7cff89-wl4bq" Nov 01 22:24:04.872921 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.872859 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-h4shp" event={"ID":"d9be2a9d-8cd7-4fc4-ab6a-8b3c7793c5bd","Type":"ContainerStarted","Data":"a0c687350a1552e1d8c92776d8f4b6d389ea42c6d61b2da3cd2e1a721e8405ab"} Nov 01 22:24:04.876651 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.876627 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5349ab21-2645-4464-a07c-d71e4503808d","Type":"ContainerStarted","Data":"ec676626594f9758c2a042a6e3d0e1baf973296bb89a38388becdf9a32aa5a69"} Nov 01 22:24:04.888876 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.888803 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-h4shp" podStartSLOduration=1.71222593 podStartE2EDuration="4.888789901s" totalImagesPullingTime="3.176563971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:04.888165785 +0000 UTC m=+13.292756418" watchObservedRunningTime="2026-11-01 22:24:04.888789901 +0000 UTC m=+13.293380531" Nov 01 22:24:04.918052 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:04.917972 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.556146793 podStartE2EDuration="8.917949542s" totalImagesPullingTime="5.349296263s" totalInitContainerRuntime="12.506486ms" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:04.914670304 +0000 UTC m=+13.319260950" watchObservedRunningTime="2026-11-01 22:24:04.917949542 +0000 UTC m=+13.322540171" Nov 01 22:24:05.125422 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:05.125306 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-777f7cff89-wl4bq"] Nov 01 22:24:05.129681 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:24:05.129652 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc81400_7732_4692_84b1_9b9cd92e13ea.slice/crio-76b47a5e098ab5faa682f3138cb0cb7af518e029a8a74c7445d3880e120af8fa WatchSource:0}: Error finding container 76b47a5e098ab5faa682f3138cb0cb7af518e029a8a74c7445d3880e120af8fa: Status 404 returned error can't find the container with id 76b47a5e098ab5faa682f3138cb0cb7af518e029a8a74c7445d3880e120af8fa Nov 01 22:24:05.885147 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:05.885117 5011 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 22:24:05.885550 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:05.885105 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-777f7cff89-wl4bq" event={"ID":"cfc81400-7732-4692-84b1-9b9cd92e13ea","Type":"ContainerStarted","Data":"76b47a5e098ab5faa682f3138cb0cb7af518e029a8a74c7445d3880e120af8fa"} Nov 01 22:24:06.453849 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:06.453794 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-h4shp" Nov 01 22:24:06.459222 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:06.459191 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-h4shp" Nov 01 22:24:07.544719 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:07.544681 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-777f7cff89-wl4bq"] Nov 01 22:24:07.573203 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:07.573164 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5fd74c69cd-dlgmp"] Nov 01 22:24:07.576975 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:07.576945 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fd74c69cd-dlgmp" Nov 01 22:24:07.588753 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:07.588723 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fd74c69cd-dlgmp"] Nov 01 22:24:07.654040 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:07.654000 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/248bc413-99de-4b80-a73e-6e342fb22587-tmp\") pod \"console-5fd74c69cd-dlgmp\" (UID: \"248bc413-99de-4b80-a73e-6e342fb22587\") " pod="openshift-console/console-5fd74c69cd-dlgmp" Nov 01 22:24:07.654213 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:07.654128 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/248bc413-99de-4b80-a73e-6e342fb22587-console-config\") pod \"console-5fd74c69cd-dlgmp\" (UID: \"248bc413-99de-4b80-a73e-6e342fb22587\") " pod="openshift-console/console-5fd74c69cd-dlgmp" Nov 01 22:24:07.654213 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:07.654173 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/248bc413-99de-4b80-a73e-6e342fb22587-console-serving-cert\") pod \"console-5fd74c69cd-dlgmp\" (UID: \"248bc413-99de-4b80-a73e-6e342fb22587\") " pod="openshift-console/console-5fd74c69cd-dlgmp" Nov 01 22:24:07.654313 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:07.654218 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq8pw\" (UniqueName: \"kubernetes.io/projected/248bc413-99de-4b80-a73e-6e342fb22587-kube-api-access-nq8pw\") pod \"console-5fd74c69cd-dlgmp\" (UID: \"248bc413-99de-4b80-a73e-6e342fb22587\") " pod="openshift-console/console-5fd74c69cd-dlgmp" Nov 01 22:24:07.654353 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:07.654303 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/248bc413-99de-4b80-a73e-6e342fb22587-oauth-serving-cert\") pod \"console-5fd74c69cd-dlgmp\" (UID: \"248bc413-99de-4b80-a73e-6e342fb22587\") " pod="openshift-console/console-5fd74c69cd-dlgmp" Nov 01 22:24:07.654389 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:07.654351 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/248bc413-99de-4b80-a73e-6e342fb22587-trusted-ca-bundle\") pod \"console-5fd74c69cd-dlgmp\" (UID: \"248bc413-99de-4b80-a73e-6e342fb22587\") " pod="openshift-console/console-5fd74c69cd-dlgmp" Nov 01 22:24:07.654428 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:07.654387 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/248bc413-99de-4b80-a73e-6e342fb22587-console-oauth-config\") pod \"console-5fd74c69cd-dlgmp\" (UID: \"248bc413-99de-4b80-a73e-6e342fb22587\") " pod="openshift-console/console-5fd74c69cd-dlgmp" Nov 01 22:24:07.654482 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:07.654427 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/248bc413-99de-4b80-a73e-6e342fb22587-service-ca\") pod \"console-5fd74c69cd-dlgmp\" (UID: \"248bc413-99de-4b80-a73e-6e342fb22587\") " pod="openshift-console/console-5fd74c69cd-dlgmp" Nov 01 22:24:07.755526 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:07.755498 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/248bc413-99de-4b80-a73e-6e342fb22587-console-config\") pod \"console-5fd74c69cd-dlgmp\" (UID: \"248bc413-99de-4b80-a73e-6e342fb22587\") " pod="openshift-console/console-5fd74c69cd-dlgmp" Nov 01 22:24:07.755735 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:07.755537 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/248bc413-99de-4b80-a73e-6e342fb22587-console-serving-cert\") pod \"console-5fd74c69cd-dlgmp\" (UID: \"248bc413-99de-4b80-a73e-6e342fb22587\") " pod="openshift-console/console-5fd74c69cd-dlgmp" Nov 01 22:24:07.755735 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:07.755618 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq8pw\" (UniqueName: \"kubernetes.io/projected/248bc413-99de-4b80-a73e-6e342fb22587-kube-api-access-nq8pw\") pod \"console-5fd74c69cd-dlgmp\" (UID: \"248bc413-99de-4b80-a73e-6e342fb22587\") " pod="openshift-console/console-5fd74c69cd-dlgmp" Nov 01 22:24:07.755735 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:07.755662 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/248bc413-99de-4b80-a73e-6e342fb22587-oauth-serving-cert\") pod \"console-5fd74c69cd-dlgmp\" (UID: \"248bc413-99de-4b80-a73e-6e342fb22587\") " pod="openshift-console/console-5fd74c69cd-dlgmp" Nov 01 22:24:07.755735 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:07.755686 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/248bc413-99de-4b80-a73e-6e342fb22587-trusted-ca-bundle\") pod \"console-5fd74c69cd-dlgmp\" (UID: \"248bc413-99de-4b80-a73e-6e342fb22587\") " pod="openshift-console/console-5fd74c69cd-dlgmp" Nov 01 22:24:07.756071 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:07.756045 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/248bc413-99de-4b80-a73e-6e342fb22587-console-oauth-config\") pod \"console-5fd74c69cd-dlgmp\" (UID: \"248bc413-99de-4b80-a73e-6e342fb22587\") " pod="openshift-console/console-5fd74c69cd-dlgmp" Nov 01 22:24:07.756153 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:07.756106 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/248bc413-99de-4b80-a73e-6e342fb22587-service-ca\") pod \"console-5fd74c69cd-dlgmp\" (UID: \"248bc413-99de-4b80-a73e-6e342fb22587\") " pod="openshift-console/console-5fd74c69cd-dlgmp" Nov 01 22:24:07.756153 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:07.756145 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/248bc413-99de-4b80-a73e-6e342fb22587-tmp\") pod \"console-5fd74c69cd-dlgmp\" (UID: \"248bc413-99de-4b80-a73e-6e342fb22587\") " pod="openshift-console/console-5fd74c69cd-dlgmp" Nov 01 22:24:07.756394 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:07.756372 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/248bc413-99de-4b80-a73e-6e342fb22587-oauth-serving-cert\") pod \"console-5fd74c69cd-dlgmp\" (UID: \"248bc413-99de-4b80-a73e-6e342fb22587\") " pod="openshift-console/console-5fd74c69cd-dlgmp" Nov 01 22:24:07.756541 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:07.756505 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/248bc413-99de-4b80-a73e-6e342fb22587-console-config\") pod \"console-5fd74c69cd-dlgmp\" (UID: \"248bc413-99de-4b80-a73e-6e342fb22587\") " pod="openshift-console/console-5fd74c69cd-dlgmp" Nov 01 22:24:07.756906 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:07.756747 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/248bc413-99de-4b80-a73e-6e342fb22587-trusted-ca-bundle\") pod \"console-5fd74c69cd-dlgmp\" (UID: \"248bc413-99de-4b80-a73e-6e342fb22587\") " pod="openshift-console/console-5fd74c69cd-dlgmp" Nov 01 22:24:07.756906 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:07.756776 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/248bc413-99de-4b80-a73e-6e342fb22587-tmp\") pod \"console-5fd74c69cd-dlgmp\" (UID: \"248bc413-99de-4b80-a73e-6e342fb22587\") " pod="openshift-console/console-5fd74c69cd-dlgmp" Nov 01 22:24:07.757073 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:07.756968 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/248bc413-99de-4b80-a73e-6e342fb22587-service-ca\") pod \"console-5fd74c69cd-dlgmp\" (UID: \"248bc413-99de-4b80-a73e-6e342fb22587\") " pod="openshift-console/console-5fd74c69cd-dlgmp" Nov 01 22:24:07.758562 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:07.758533 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/248bc413-99de-4b80-a73e-6e342fb22587-console-serving-cert\") pod \"console-5fd74c69cd-dlgmp\" (UID: \"248bc413-99de-4b80-a73e-6e342fb22587\") " pod="openshift-console/console-5fd74c69cd-dlgmp" Nov 01 22:24:07.759192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:07.759167 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/248bc413-99de-4b80-a73e-6e342fb22587-console-oauth-config\") pod \"console-5fd74c69cd-dlgmp\" (UID: \"248bc413-99de-4b80-a73e-6e342fb22587\") " pod="openshift-console/console-5fd74c69cd-dlgmp" Nov 01 22:24:07.763382 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:07.763319 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq8pw\" (UniqueName: \"kubernetes.io/projected/248bc413-99de-4b80-a73e-6e342fb22587-kube-api-access-nq8pw\") pod \"console-5fd74c69cd-dlgmp\" (UID: \"248bc413-99de-4b80-a73e-6e342fb22587\") " pod="openshift-console/console-5fd74c69cd-dlgmp" Nov 01 22:24:07.895668 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:07.895582 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fd74c69cd-dlgmp" Nov 01 22:24:08.180535 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:08.180500 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fd74c69cd-dlgmp"] Nov 01 22:24:08.190170 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:24:08.190123 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod248bc413_99de_4b80_a73e_6e342fb22587.slice/crio-cae9ff1fcf85a91cb7c03f9f1a8dafa0170fe28900a52ca314eb3872588b0483 WatchSource:0}: Error finding container cae9ff1fcf85a91cb7c03f9f1a8dafa0170fe28900a52ca314eb3872588b0483: Status 404 returned error can't find the container with id cae9ff1fcf85a91cb7c03f9f1a8dafa0170fe28900a52ca314eb3872588b0483 Nov 01 22:24:08.898650 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:08.898560 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-777f7cff89-wl4bq" event={"ID":"cfc81400-7732-4692-84b1-9b9cd92e13ea","Type":"ContainerStarted","Data":"304b39d11a9a97279ddc2f5c6e30e80e032a95880bd311f947682c6819dfb127"} Nov 01 22:24:08.900598 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:08.900569 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fd74c69cd-dlgmp" event={"ID":"248bc413-99de-4b80-a73e-6e342fb22587","Type":"ContainerStarted","Data":"13359b44df6cf101cbe21cebc18b6c7066150d1b1c8d676a8e325df4018dd6e9"} Nov 01 22:24:08.900709 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:08.900612 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fd74c69cd-dlgmp" event={"ID":"248bc413-99de-4b80-a73e-6e342fb22587","Type":"ContainerStarted","Data":"cae9ff1fcf85a91cb7c03f9f1a8dafa0170fe28900a52ca314eb3872588b0483"} Nov 01 22:24:08.915480 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:08.915419 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-777f7cff89-wl4bq" podStartSLOduration=1.619653344 podStartE2EDuration="4.915401323s" totalImagesPullingTime="3.295747979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:08.913904197 +0000 UTC m=+17.318494830" watchObservedRunningTime="2026-11-01 22:24:08.915401323 +0000 UTC m=+17.319991956" Nov 01 22:24:08.920096 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:08.920071 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6h899" Nov 01 22:24:08.929628 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:08.929571 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5fd74c69cd-dlgmp" podStartSLOduration=1.536526019 podStartE2EDuration="1.929557482s" totalImagesPullingTime="393.031463ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:08.929119635 +0000 UTC m=+17.333710268" watchObservedRunningTime="2026-11-01 22:24:08.929557482 +0000 UTC m=+17.334148114" Nov 01 22:24:13.346484 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:13.346452 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tmhm2" Nov 01 22:24:13.696577 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:13.696536 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-sk8f6-sf7bv"] Nov 01 22:24:13.697334 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:13.697103 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-sk8f6-sf7bv" podUID="3880904a-615b-44ac-9a2d-7b0f7ad31e0c" containerName="insights-gathering" containerID="cri-o://fe116260e63ccb4c5c548d63bebf6b667b07d5f9b4ce7deaa367b417a7fcd04d" gracePeriod=30 Nov 01 22:24:13.919765 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:13.919729 5011 generic.go:356] "Generic (PLEG): container finished" podID="3880904a-615b-44ac-9a2d-7b0f7ad31e0c" containerID="fe116260e63ccb4c5c548d63bebf6b667b07d5f9b4ce7deaa367b417a7fcd04d" exitCode=2 Nov 01 22:24:13.919948 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:13.919814 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-sk8f6-sf7bv" event={"ID":"3880904a-615b-44ac-9a2d-7b0f7ad31e0c","Type":"ContainerDied","Data":"fe116260e63ccb4c5c548d63bebf6b667b07d5f9b4ce7deaa367b417a7fcd04d"} Nov 01 22:24:13.969468 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:13.967442 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-sk8f6-sf7bv" Nov 01 22:24:14.116545 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:14.116508 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-trusted-ca-bundle\") pod \"3880904a-615b-44ac-9a2d-7b0f7ad31e0c\" (UID: \"3880904a-615b-44ac-9a2d-7b0f7ad31e0c\") " Nov 01 22:24:14.116737 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:14.116587 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-archives-path\") pod \"3880904a-615b-44ac-9a2d-7b0f7ad31e0c\" (UID: \"3880904a-615b-44ac-9a2d-7b0f7ad31e0c\") " Nov 01 22:24:14.116737 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:14.116614 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-kube-api-access-8ndlv\" (UniqueName: \"kubernetes.io/projected/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-kube-api-access-8ndlv\") pod \"3880904a-615b-44ac-9a2d-7b0f7ad31e0c\" (UID: \"3880904a-615b-44ac-9a2d-7b0f7ad31e0c\") " Nov 01 22:24:14.116737 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:14.116630 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-service-ca-bundle\") pod \"3880904a-615b-44ac-9a2d-7b0f7ad31e0c\" (UID: \"3880904a-615b-44ac-9a2d-7b0f7ad31e0c\") " Nov 01 22:24:14.116963 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:14.116927 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-trusted-ca-bundle" pod "3880904a-615b-44ac-9a2d-7b0f7ad31e0c" (UID: "3880904a-615b-44ac-9a2d-7b0f7ad31e0c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:14.116963 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:14.116940 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-archives-path" pod "3880904a-615b-44ac-9a2d-7b0f7ad31e0c" (UID: "3880904a-615b-44ac-9a2d-7b0f7ad31e0c"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:24:14.117129 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:14.117033 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-service-ca-bundle" pod "3880904a-615b-44ac-9a2d-7b0f7ad31e0c" (UID: "3880904a-615b-44ac-9a2d-7b0f7ad31e0c"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:14.120667 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:14.120628 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-kube-api-access-8ndlv" pod "3880904a-615b-44ac-9a2d-7b0f7ad31e0c" (UID: "3880904a-615b-44ac-9a2d-7b0f7ad31e0c"). InnerVolumeSpecName "kube-api-access-8ndlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:24:14.217808 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:14.217735 5011 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-archives-path\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:14.217808 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:14.217769 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ndlv\" (UniqueName: \"kubernetes.io/projected/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-kube-api-access-8ndlv\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:14.217808 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:14.217780 5011 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-service-ca-bundle\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:14.217808 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:14.217789 5011 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3880904a-615b-44ac-9a2d-7b0f7ad31e0c-trusted-ca-bundle\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:14.853182 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:14.853145 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-777f7cff89-wl4bq" Nov 01 22:24:14.924801 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:14.924775 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-sk8f6-sf7bv" Nov 01 22:24:14.924958 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:14.924766 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-sk8f6-sf7bv" event={"ID":"3880904a-615b-44ac-9a2d-7b0f7ad31e0c","Type":"ContainerDied","Data":"7385d8d2150889036d38eaa7fc4e655bf36d9fb0ee541e0edca952a91e2a79c7"} Nov 01 22:24:14.924958 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:14.924925 5011 scope.go:122] "RemoveContainer" containerID="fe116260e63ccb4c5c548d63bebf6b667b07d5f9b4ce7deaa367b417a7fcd04d" Nov 01 22:24:14.951017 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:14.950985 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-sk8f6-sf7bv"] Nov 01 22:24:14.955148 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:14.955126 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-sk8f6-sf7bv"] Nov 01 22:24:15.411042 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:15.410996 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq" Nov 01 22:24:15.743685 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:15.743649 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3880904a-615b-44ac-9a2d-7b0f7ad31e0c" path="/var/lib/kubelet/pods/3880904a-615b-44ac-9a2d-7b0f7ad31e0c/volumes" Nov 01 22:24:16.957979 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:24:16.957945 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:24:17.895905 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:17.895864 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fd74c69cd-dlgmp" Nov 01 22:24:17.896111 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:17.895929 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5fd74c69cd-dlgmp" Nov 01 22:24:17.900804 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:17.900784 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5fd74c69cd-dlgmp" Nov 01 22:24:17.940203 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:17.940175 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5fd74c69cd-dlgmp" Nov 01 22:24:20.942286 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:20.942254 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq" Nov 01 22:24:27.800314 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:27.800285 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cfnmp" Nov 01 22:24:33.920108 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:33.920037 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-777f7cff89-wl4bq" podUID="cfc81400-7732-4692-84b1-9b9cd92e13ea" containerName="console" containerID="cri-o://304b39d11a9a97279ddc2f5c6e30e80e032a95880bd311f947682c6819dfb127" gracePeriod=15 Nov 01 22:24:34.161535 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:34.161506 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-777f7cff89-wl4bq" Nov 01 22:24:34.183097 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:34.183015 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfc81400-7732-4692-84b1-9b9cd92e13ea-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfc81400-7732-4692-84b1-9b9cd92e13ea-service-ca\") pod \"cfc81400-7732-4692-84b1-9b9cd92e13ea\" (UID: \"cfc81400-7732-4692-84b1-9b9cd92e13ea\") " Nov 01 22:24:34.183097 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:34.183060 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfc81400-7732-4692-84b1-9b9cd92e13ea-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cfc81400-7732-4692-84b1-9b9cd92e13ea-oauth-serving-cert\") pod \"cfc81400-7732-4692-84b1-9b9cd92e13ea\" (UID: \"cfc81400-7732-4692-84b1-9b9cd92e13ea\") " Nov 01 22:24:34.183097 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:34.183093 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfc81400-7732-4692-84b1-9b9cd92e13ea-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cfc81400-7732-4692-84b1-9b9cd92e13ea-console-serving-cert\") pod \"cfc81400-7732-4692-84b1-9b9cd92e13ea\" (UID: \"cfc81400-7732-4692-84b1-9b9cd92e13ea\") " Nov 01 22:24:34.183352 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:34.183139 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfc81400-7732-4692-84b1-9b9cd92e13ea-kube-api-access-22blp\" (UniqueName: \"kubernetes.io/projected/cfc81400-7732-4692-84b1-9b9cd92e13ea-kube-api-access-22blp\") pod \"cfc81400-7732-4692-84b1-9b9cd92e13ea\" (UID: \"cfc81400-7732-4692-84b1-9b9cd92e13ea\") " Nov 01 22:24:34.183352 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:34.183183 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfc81400-7732-4692-84b1-9b9cd92e13ea-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfc81400-7732-4692-84b1-9b9cd92e13ea-trusted-ca-bundle\") pod \"cfc81400-7732-4692-84b1-9b9cd92e13ea\" (UID: \"cfc81400-7732-4692-84b1-9b9cd92e13ea\") " Nov 01 22:24:34.183352 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:34.183205 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfc81400-7732-4692-84b1-9b9cd92e13ea-console-config\" (UniqueName: \"kubernetes.io/configmap/cfc81400-7732-4692-84b1-9b9cd92e13ea-console-config\") pod \"cfc81400-7732-4692-84b1-9b9cd92e13ea\" (UID: \"cfc81400-7732-4692-84b1-9b9cd92e13ea\") " Nov 01 22:24:34.183352 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:34.183232 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfc81400-7732-4692-84b1-9b9cd92e13ea-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cfc81400-7732-4692-84b1-9b9cd92e13ea-console-oauth-config\") pod \"cfc81400-7732-4692-84b1-9b9cd92e13ea\" (UID: \"cfc81400-7732-4692-84b1-9b9cd92e13ea\") " Nov 01 22:24:34.183352 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:34.183298 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfc81400-7732-4692-84b1-9b9cd92e13ea-tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfc81400-7732-4692-84b1-9b9cd92e13ea-tmp\") pod \"cfc81400-7732-4692-84b1-9b9cd92e13ea\" (UID: \"cfc81400-7732-4692-84b1-9b9cd92e13ea\") " Nov 01 22:24:34.183595 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:34.183539 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfc81400-7732-4692-84b1-9b9cd92e13ea-oauth-serving-cert" pod "cfc81400-7732-4692-84b1-9b9cd92e13ea" (UID: "cfc81400-7732-4692-84b1-9b9cd92e13ea"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:34.183595 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:34.183554 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfc81400-7732-4692-84b1-9b9cd92e13ea-service-ca" pod "cfc81400-7732-4692-84b1-9b9cd92e13ea" (UID: "cfc81400-7732-4692-84b1-9b9cd92e13ea"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:34.183709 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:34.183687 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc81400-7732-4692-84b1-9b9cd92e13ea-tmp" pod "cfc81400-7732-4692-84b1-9b9cd92e13ea" (UID: "cfc81400-7732-4692-84b1-9b9cd92e13ea"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:24:34.183925 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:34.183901 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfc81400-7732-4692-84b1-9b9cd92e13ea-console-config" pod "cfc81400-7732-4692-84b1-9b9cd92e13ea" (UID: "cfc81400-7732-4692-84b1-9b9cd92e13ea"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:34.184021 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:34.183915 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfc81400-7732-4692-84b1-9b9cd92e13ea-trusted-ca-bundle" pod "cfc81400-7732-4692-84b1-9b9cd92e13ea" (UID: "cfc81400-7732-4692-84b1-9b9cd92e13ea"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:34.185979 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:34.185949 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfc81400-7732-4692-84b1-9b9cd92e13ea-console-oauth-config" pod "cfc81400-7732-4692-84b1-9b9cd92e13ea" (UID: "cfc81400-7732-4692-84b1-9b9cd92e13ea"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:24:34.186165 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:34.186137 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfc81400-7732-4692-84b1-9b9cd92e13ea-console-serving-cert" pod "cfc81400-7732-4692-84b1-9b9cd92e13ea" (UID: "cfc81400-7732-4692-84b1-9b9cd92e13ea"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:24:34.186292 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:34.186221 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc81400-7732-4692-84b1-9b9cd92e13ea-kube-api-access-22blp" pod "cfc81400-7732-4692-84b1-9b9cd92e13ea" (UID: "cfc81400-7732-4692-84b1-9b9cd92e13ea"). InnerVolumeSpecName "kube-api-access-22blp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:24:34.284781 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:34.284746 5011 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfc81400-7732-4692-84b1-9b9cd92e13ea-tmp\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:34.284781 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:34.284776 5011 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cfc81400-7732-4692-84b1-9b9cd92e13ea-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:34.284781 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:34.284788 5011 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cfc81400-7732-4692-84b1-9b9cd92e13ea-oauth-serving-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:34.285044 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:34.284799 5011 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cfc81400-7732-4692-84b1-9b9cd92e13ea-console-serving-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:34.285044 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:34.284808 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22blp\" (UniqueName: \"kubernetes.io/projected/cfc81400-7732-4692-84b1-9b9cd92e13ea-kube-api-access-22blp\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:34.285044 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:34.284817 5011 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfc81400-7732-4692-84b1-9b9cd92e13ea-trusted-ca-bundle\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:34.285044 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:34.284826 5011 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cfc81400-7732-4692-84b1-9b9cd92e13ea-console-config\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:34.285044 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:34.284858 5011 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cfc81400-7732-4692-84b1-9b9cd92e13ea-console-oauth-config\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:34.996129 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:34.996093 5011 generic.go:356] "Generic (PLEG): container finished" podID="cfc81400-7732-4692-84b1-9b9cd92e13ea" containerID="304b39d11a9a97279ddc2f5c6e30e80e032a95880bd311f947682c6819dfb127" exitCode=0 Nov 01 22:24:34.996585 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:34.996135 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-777f7cff89-wl4bq" event={"ID":"cfc81400-7732-4692-84b1-9b9cd92e13ea","Type":"ContainerDied","Data":"304b39d11a9a97279ddc2f5c6e30e80e032a95880bd311f947682c6819dfb127"} Nov 01 22:24:34.996585 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:34.996156 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-777f7cff89-wl4bq" Nov 01 22:24:34.996585 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:34.996201 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-777f7cff89-wl4bq" event={"ID":"cfc81400-7732-4692-84b1-9b9cd92e13ea","Type":"ContainerDied","Data":"76b47a5e098ab5faa682f3138cb0cb7af518e029a8a74c7445d3880e120af8fa"} Nov 01 22:24:34.996585 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:34.996242 5011 scope.go:122] "RemoveContainer" containerID="304b39d11a9a97279ddc2f5c6e30e80e032a95880bd311f947682c6819dfb127" Nov 01 22:24:35.004613 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:35.004597 5011 scope.go:122] "RemoveContainer" containerID="304b39d11a9a97279ddc2f5c6e30e80e032a95880bd311f947682c6819dfb127" Nov 01 22:24:35.004901 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:24:35.004881 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"304b39d11a9a97279ddc2f5c6e30e80e032a95880bd311f947682c6819dfb127\": container with ID starting with 304b39d11a9a97279ddc2f5c6e30e80e032a95880bd311f947682c6819dfb127 not found: ID does not exist" containerID="304b39d11a9a97279ddc2f5c6e30e80e032a95880bd311f947682c6819dfb127" Nov 01 22:24:35.004953 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:35.004910 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"304b39d11a9a97279ddc2f5c6e30e80e032a95880bd311f947682c6819dfb127"} err="failed to get container status \"304b39d11a9a97279ddc2f5c6e30e80e032a95880bd311f947682c6819dfb127\": rpc error: code = NotFound desc = could not find container \"304b39d11a9a97279ddc2f5c6e30e80e032a95880bd311f947682c6819dfb127\": container with ID starting with 304b39d11a9a97279ddc2f5c6e30e80e032a95880bd311f947682c6819dfb127 not found: ID does not exist" Nov 01 22:24:35.015953 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:35.015927 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-777f7cff89-wl4bq"] Nov 01 22:24:35.019467 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:35.019446 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-777f7cff89-wl4bq"] Nov 01 22:24:35.417042 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:35.416967 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq" Nov 01 22:24:35.420740 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:35.420715 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-66f8d6ccdb-f9xnq" Nov 01 22:24:35.744305 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:35.744274 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfc81400-7732-4692-84b1-9b9cd92e13ea" path="/var/lib/kubelet/pods/cfc81400-7732-4692-84b1-9b9cd92e13ea/volumes" Nov 01 22:24:40.389030 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:40.388997 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 22:24:40.389433 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:40.389302 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5349ab21-2645-4464-a07c-d71e4503808d" containerName="alertmanager" containerID="cri-o://a7223bd264b3d6e9b9360c895ce6d4adb5d4d9ea532ae55d7892b6c2896c00d2" gracePeriod=120 Nov 01 22:24:40.389433 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:40.389363 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5349ab21-2645-4464-a07c-d71e4503808d" containerName="kube-rbac-proxy-metric" containerID="cri-o://93e0505dc7ebcad2d5e61ba5ed9d5c84fd4a491847687ecd02182316d836ecba" gracePeriod=120 Nov 01 22:24:40.389433 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:40.389403 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5349ab21-2645-4464-a07c-d71e4503808d" containerName="prom-label-proxy" containerID="cri-o://ec676626594f9758c2a042a6e3d0e1baf973296bb89a38388becdf9a32aa5a69" gracePeriod=120 Nov 01 22:24:40.389568 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:40.389430 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5349ab21-2645-4464-a07c-d71e4503808d" containerName="kube-rbac-proxy-web" containerID="cri-o://8873b887bb93503b0e9ac0ca4e5d30e30ef27bfc0a1844a03d250cfbdba3f5e9" gracePeriod=120 Nov 01 22:24:40.389568 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:40.389458 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5349ab21-2645-4464-a07c-d71e4503808d" containerName="config-reloader" containerID="cri-o://acac8477146c8e4ec5bddf7d1684ff493357d1a02d26748cd5540211f41a5a51" gracePeriod=120 Nov 01 22:24:40.389568 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:40.389530 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5349ab21-2645-4464-a07c-d71e4503808d" containerName="kube-rbac-proxy" containerID="cri-o://6bc3a0740c090bbc8a3983f168e06b3cf7a0696dd1f9c76aae0c662e74f282c5" gracePeriod=120 Nov 01 22:24:41.021440 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.021406 5011 generic.go:356] "Generic (PLEG): container finished" podID="5349ab21-2645-4464-a07c-d71e4503808d" containerID="ec676626594f9758c2a042a6e3d0e1baf973296bb89a38388becdf9a32aa5a69" exitCode=0 Nov 01 22:24:41.021440 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.021431 5011 generic.go:356] "Generic (PLEG): container finished" podID="5349ab21-2645-4464-a07c-d71e4503808d" containerID="6bc3a0740c090bbc8a3983f168e06b3cf7a0696dd1f9c76aae0c662e74f282c5" exitCode=0 Nov 01 22:24:41.021440 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.021438 5011 generic.go:356] "Generic (PLEG): container finished" podID="5349ab21-2645-4464-a07c-d71e4503808d" containerID="8873b887bb93503b0e9ac0ca4e5d30e30ef27bfc0a1844a03d250cfbdba3f5e9" exitCode=0 Nov 01 22:24:41.021440 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.021444 5011 generic.go:356] "Generic (PLEG): container finished" podID="5349ab21-2645-4464-a07c-d71e4503808d" containerID="acac8477146c8e4ec5bddf7d1684ff493357d1a02d26748cd5540211f41a5a51" exitCode=0 Nov 01 22:24:41.021440 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.021448 5011 generic.go:356] "Generic (PLEG): container finished" podID="5349ab21-2645-4464-a07c-d71e4503808d" containerID="a7223bd264b3d6e9b9360c895ce6d4adb5d4d9ea532ae55d7892b6c2896c00d2" exitCode=0 Nov 01 22:24:41.021737 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.021473 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5349ab21-2645-4464-a07c-d71e4503808d","Type":"ContainerDied","Data":"ec676626594f9758c2a042a6e3d0e1baf973296bb89a38388becdf9a32aa5a69"} Nov 01 22:24:41.021737 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.021534 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5349ab21-2645-4464-a07c-d71e4503808d","Type":"ContainerDied","Data":"6bc3a0740c090bbc8a3983f168e06b3cf7a0696dd1f9c76aae0c662e74f282c5"} Nov 01 22:24:41.021737 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.021561 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5349ab21-2645-4464-a07c-d71e4503808d","Type":"ContainerDied","Data":"8873b887bb93503b0e9ac0ca4e5d30e30ef27bfc0a1844a03d250cfbdba3f5e9"} Nov 01 22:24:41.021737 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.021585 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5349ab21-2645-4464-a07c-d71e4503808d","Type":"ContainerDied","Data":"acac8477146c8e4ec5bddf7d1684ff493357d1a02d26748cd5540211f41a5a51"} Nov 01 22:24:41.021737 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.021609 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5349ab21-2645-4464-a07c-d71e4503808d","Type":"ContainerDied","Data":"a7223bd264b3d6e9b9360c895ce6d4adb5d4d9ea532ae55d7892b6c2896c00d2"} Nov 01 22:24:41.632078 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.632053 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:41.739650 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.739563 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-secret-alertmanager-kube-rbac-proxy\") pod \"5349ab21-2645-4464-a07c-d71e4503808d\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " Nov 01 22:24:41.739650 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.739624 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-secret-alertmanager-main-tls\") pod \"5349ab21-2645-4464-a07c-d71e4503808d\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " Nov 01 22:24:41.739911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.739668 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5349ab21-2645-4464-a07c-d71e4503808d-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5349ab21-2645-4464-a07c-d71e4503808d-alertmanager-main-db\") pod \"5349ab21-2645-4464-a07c-d71e4503808d\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " Nov 01 22:24:41.739911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.739699 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-secret-alertmanager-kube-rbac-proxy-web\") pod \"5349ab21-2645-4464-a07c-d71e4503808d\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " Nov 01 22:24:41.739911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.739735 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-config-volume\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-config-volume\") pod \"5349ab21-2645-4464-a07c-d71e4503808d\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " Nov 01 22:24:41.739911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.739777 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-web-config\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-web-config\") pod \"5349ab21-2645-4464-a07c-d71e4503808d\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " Nov 01 22:24:41.739911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.739821 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"5349ab21-2645-4464-a07c-d71e4503808d\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " Nov 01 22:24:41.739911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.739883 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5349ab21-2645-4464-a07c-d71e4503808d-tls-assets\" (UniqueName: \"kubernetes.io/projected/5349ab21-2645-4464-a07c-d71e4503808d-tls-assets\") pod \"5349ab21-2645-4464-a07c-d71e4503808d\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " Nov 01 22:24:41.739911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.739910 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-cluster-tls-config\") pod \"5349ab21-2645-4464-a07c-d71e4503808d\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " Nov 01 22:24:41.740274 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.739942 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5349ab21-2645-4464-a07c-d71e4503808d-kube-api-access-mgxh6\" (UniqueName: \"kubernetes.io/projected/5349ab21-2645-4464-a07c-d71e4503808d-kube-api-access-mgxh6\") pod \"5349ab21-2645-4464-a07c-d71e4503808d\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " Nov 01 22:24:41.740274 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.739976 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5349ab21-2645-4464-a07c-d71e4503808d-config-out\" (UniqueName: \"kubernetes.io/empty-dir/5349ab21-2645-4464-a07c-d71e4503808d-config-out\") pod \"5349ab21-2645-4464-a07c-d71e4503808d\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " Nov 01 22:24:41.740274 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.740010 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5349ab21-2645-4464-a07c-d71e4503808d-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5349ab21-2645-4464-a07c-d71e4503808d-alertmanager-trusted-ca-bundle\") pod \"5349ab21-2645-4464-a07c-d71e4503808d\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " Nov 01 22:24:41.740274 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.740037 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5349ab21-2645-4464-a07c-d71e4503808d-alertmanager-main-db" pod "5349ab21-2645-4464-a07c-d71e4503808d" (UID: "5349ab21-2645-4464-a07c-d71e4503808d"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:24:41.740274 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.740051 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5349ab21-2645-4464-a07c-d71e4503808d-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5349ab21-2645-4464-a07c-d71e4503808d-metrics-client-ca\") pod \"5349ab21-2645-4464-a07c-d71e4503808d\" (UID: \"5349ab21-2645-4464-a07c-d71e4503808d\") " Nov 01 22:24:41.740274 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.740265 5011 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5349ab21-2645-4464-a07c-d71e4503808d-alertmanager-main-db\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:41.742776 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.742748 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-secret-alertmanager-main-tls" pod "5349ab21-2645-4464-a07c-d71e4503808d" (UID: "5349ab21-2645-4464-a07c-d71e4503808d"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:24:41.742930 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.742815 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5349ab21-2645-4464-a07c-d71e4503808d-tls-assets" pod "5349ab21-2645-4464-a07c-d71e4503808d" (UID: "5349ab21-2645-4464-a07c-d71e4503808d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:24:41.742930 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.742856 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-secret-alertmanager-kube-rbac-proxy" pod "5349ab21-2645-4464-a07c-d71e4503808d" (UID: "5349ab21-2645-4464-a07c-d71e4503808d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:24:41.742930 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.742873 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-secret-alertmanager-kube-rbac-proxy-web" pod "5349ab21-2645-4464-a07c-d71e4503808d" (UID: "5349ab21-2645-4464-a07c-d71e4503808d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:24:41.743231 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.743182 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5349ab21-2645-4464-a07c-d71e4503808d-alertmanager-trusted-ca-bundle" pod "5349ab21-2645-4464-a07c-d71e4503808d" (UID: "5349ab21-2645-4464-a07c-d71e4503808d"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:41.743350 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.743252 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5349ab21-2645-4464-a07c-d71e4503808d-metrics-client-ca" pod "5349ab21-2645-4464-a07c-d71e4503808d" (UID: "5349ab21-2645-4464-a07c-d71e4503808d"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:41.743728 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.743693 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5349ab21-2645-4464-a07c-d71e4503808d-kube-api-access-mgxh6" pod "5349ab21-2645-4464-a07c-d71e4503808d" (UID: "5349ab21-2645-4464-a07c-d71e4503808d"). InnerVolumeSpecName "kube-api-access-mgxh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:24:41.743900 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.743882 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5349ab21-2645-4464-a07c-d71e4503808d-config-out" pod "5349ab21-2645-4464-a07c-d71e4503808d" (UID: "5349ab21-2645-4464-a07c-d71e4503808d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:24:41.745552 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.745532 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-config-volume" pod "5349ab21-2645-4464-a07c-d71e4503808d" (UID: "5349ab21-2645-4464-a07c-d71e4503808d"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:24:41.745656 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.745628 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-secret-alertmanager-kube-rbac-proxy-metric" pod "5349ab21-2645-4464-a07c-d71e4503808d" (UID: "5349ab21-2645-4464-a07c-d71e4503808d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:24:41.749499 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.749472 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-cluster-tls-config" pod "5349ab21-2645-4464-a07c-d71e4503808d" (UID: "5349ab21-2645-4464-a07c-d71e4503808d"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:24:41.755207 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.755129 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-web-config" pod "5349ab21-2645-4464-a07c-d71e4503808d" (UID: "5349ab21-2645-4464-a07c-d71e4503808d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:24:41.840637 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.840607 5011 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-web-config\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:41.840637 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.840635 5011 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:41.840860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.840649 5011 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5349ab21-2645-4464-a07c-d71e4503808d-tls-assets\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:41.840860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.840666 5011 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-cluster-tls-config\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:41.840860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.840680 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgxh6\" (UniqueName: \"kubernetes.io/projected/5349ab21-2645-4464-a07c-d71e4503808d-kube-api-access-mgxh6\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:41.840860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.840694 5011 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5349ab21-2645-4464-a07c-d71e4503808d-config-out\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:41.840860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.840708 5011 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5349ab21-2645-4464-a07c-d71e4503808d-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:41.840860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.840721 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5349ab21-2645-4464-a07c-d71e4503808d-metrics-client-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:41.840860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.840730 5011 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:41.840860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.840745 5011 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-secret-alertmanager-main-tls\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:41.840860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.840754 5011 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:41.840860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:41.840763 5011 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5349ab21-2645-4464-a07c-d71e4503808d-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:42.031304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.031199 5011 generic.go:356] "Generic (PLEG): container finished" podID="5349ab21-2645-4464-a07c-d71e4503808d" containerID="93e0505dc7ebcad2d5e61ba5ed9d5c84fd4a491847687ecd02182316d836ecba" exitCode=0 Nov 01 22:24:42.031304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.031275 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5349ab21-2645-4464-a07c-d71e4503808d","Type":"ContainerDied","Data":"93e0505dc7ebcad2d5e61ba5ed9d5c84fd4a491847687ecd02182316d836ecba"} Nov 01 22:24:42.031482 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.031284 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.031482 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.031364 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5349ab21-2645-4464-a07c-d71e4503808d","Type":"ContainerDied","Data":"d36d7430cc118bd77f441e3cd452ae5c6c30a61deef5c59bf02d915b7855e7c9"} Nov 01 22:24:42.031482 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.031397 5011 scope.go:122] "RemoveContainer" containerID="ec676626594f9758c2a042a6e3d0e1baf973296bb89a38388becdf9a32aa5a69" Nov 01 22:24:42.039715 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.039700 5011 scope.go:122] "RemoveContainer" containerID="93e0505dc7ebcad2d5e61ba5ed9d5c84fd4a491847687ecd02182316d836ecba" Nov 01 22:24:42.047208 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.047181 5011 scope.go:122] "RemoveContainer" containerID="6bc3a0740c090bbc8a3983f168e06b3cf7a0696dd1f9c76aae0c662e74f282c5" Nov 01 22:24:42.053756 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.053734 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 22:24:42.054140 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.054116 5011 scope.go:122] "RemoveContainer" containerID="8873b887bb93503b0e9ac0ca4e5d30e30ef27bfc0a1844a03d250cfbdba3f5e9" Nov 01 22:24:42.056988 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.056968 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 22:24:42.061396 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.061378 5011 scope.go:122] "RemoveContainer" containerID="acac8477146c8e4ec5bddf7d1684ff493357d1a02d26748cd5540211f41a5a51" Nov 01 22:24:42.069815 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.069799 5011 scope.go:122] "RemoveContainer" containerID="a7223bd264b3d6e9b9360c895ce6d4adb5d4d9ea532ae55d7892b6c2896c00d2" Nov 01 22:24:42.077236 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.077220 5011 scope.go:122] "RemoveContainer" containerID="d331363f4284d8fd77ab657eafb70be3c11dfa2ad3b3e96a48a46020fbc2b126" Nov 01 22:24:42.081696 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.081676 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 22:24:42.081963 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.081950 5011 cpu_manager.go:414] "containerMap: removing container" podUID="3880904a-615b-44ac-9a2d-7b0f7ad31e0c" containerName="insights-gathering" Nov 01 22:24:42.082019 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.081965 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3880904a-615b-44ac-9a2d-7b0f7ad31e0c" containerName="insights-gathering" Nov 01 22:24:42.082019 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.081976 5011 cpu_manager.go:414] "containerMap: removing container" podUID="5349ab21-2645-4464-a07c-d71e4503808d" containerName="kube-rbac-proxy-web" Nov 01 22:24:42.082019 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.081982 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5349ab21-2645-4464-a07c-d71e4503808d" containerName="kube-rbac-proxy-web" Nov 01 22:24:42.082019 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.081989 5011 cpu_manager.go:414] "containerMap: removing container" podUID="5349ab21-2645-4464-a07c-d71e4503808d" containerName="alertmanager" Nov 01 22:24:42.082019 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.081993 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5349ab21-2645-4464-a07c-d71e4503808d" containerName="alertmanager" Nov 01 22:24:42.082019 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.082000 5011 cpu_manager.go:414] "containerMap: removing container" podUID="5349ab21-2645-4464-a07c-d71e4503808d" containerName="kube-rbac-proxy" Nov 01 22:24:42.082019 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.082004 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5349ab21-2645-4464-a07c-d71e4503808d" containerName="kube-rbac-proxy" Nov 01 22:24:42.082019 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.082011 5011 cpu_manager.go:414] "containerMap: removing container" podUID="5349ab21-2645-4464-a07c-d71e4503808d" containerName="kube-rbac-proxy-metric" Nov 01 22:24:42.082019 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.082015 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5349ab21-2645-4464-a07c-d71e4503808d" containerName="kube-rbac-proxy-metric" Nov 01 22:24:42.082278 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.082031 5011 cpu_manager.go:414] "containerMap: removing container" podUID="5349ab21-2645-4464-a07c-d71e4503808d" containerName="config-reloader" Nov 01 22:24:42.082278 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.082035 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5349ab21-2645-4464-a07c-d71e4503808d" containerName="config-reloader" Nov 01 22:24:42.082278 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.082043 5011 cpu_manager.go:414] "containerMap: removing container" podUID="5349ab21-2645-4464-a07c-d71e4503808d" containerName="init-config-reloader" Nov 01 22:24:42.082278 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.082047 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5349ab21-2645-4464-a07c-d71e4503808d" containerName="init-config-reloader" Nov 01 22:24:42.082278 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.082054 5011 cpu_manager.go:414] "containerMap: removing container" podUID="cfc81400-7732-4692-84b1-9b9cd92e13ea" containerName="console" Nov 01 22:24:42.082278 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.082058 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfc81400-7732-4692-84b1-9b9cd92e13ea" containerName="console" Nov 01 22:24:42.082278 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.082066 5011 cpu_manager.go:414] "containerMap: removing container" podUID="5349ab21-2645-4464-a07c-d71e4503808d" containerName="prom-label-proxy" Nov 01 22:24:42.082278 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.082070 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5349ab21-2645-4464-a07c-d71e4503808d" containerName="prom-label-proxy" Nov 01 22:24:42.082278 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.082139 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="5349ab21-2645-4464-a07c-d71e4503808d" containerName="prom-label-proxy" Nov 01 22:24:42.082278 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.082150 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="3880904a-615b-44ac-9a2d-7b0f7ad31e0c" containerName="insights-gathering" Nov 01 22:24:42.082278 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.082156 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="5349ab21-2645-4464-a07c-d71e4503808d" containerName="config-reloader" Nov 01 22:24:42.082278 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.082161 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="5349ab21-2645-4464-a07c-d71e4503808d" containerName="kube-rbac-proxy-web" Nov 01 22:24:42.082278 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.082168 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="5349ab21-2645-4464-a07c-d71e4503808d" containerName="alertmanager" Nov 01 22:24:42.082278 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.082174 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="5349ab21-2645-4464-a07c-d71e4503808d" containerName="kube-rbac-proxy" Nov 01 22:24:42.082278 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.082178 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="5349ab21-2645-4464-a07c-d71e4503808d" containerName="kube-rbac-proxy-metric" Nov 01 22:24:42.082278 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.082183 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfc81400-7732-4692-84b1-9b9cd92e13ea" containerName="console" Nov 01 22:24:42.084756 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.084737 5011 scope.go:122] "RemoveContainer" containerID="ec676626594f9758c2a042a6e3d0e1baf973296bb89a38388becdf9a32aa5a69" Nov 01 22:24:42.085052 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:24:42.085033 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec676626594f9758c2a042a6e3d0e1baf973296bb89a38388becdf9a32aa5a69\": container with ID starting with ec676626594f9758c2a042a6e3d0e1baf973296bb89a38388becdf9a32aa5a69 not found: ID does not exist" containerID="ec676626594f9758c2a042a6e3d0e1baf973296bb89a38388becdf9a32aa5a69" Nov 01 22:24:42.085119 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.085065 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec676626594f9758c2a042a6e3d0e1baf973296bb89a38388becdf9a32aa5a69"} err="failed to get container status \"ec676626594f9758c2a042a6e3d0e1baf973296bb89a38388becdf9a32aa5a69\": rpc error: code = NotFound desc = could not find container \"ec676626594f9758c2a042a6e3d0e1baf973296bb89a38388becdf9a32aa5a69\": container with ID starting with ec676626594f9758c2a042a6e3d0e1baf973296bb89a38388becdf9a32aa5a69 not found: ID does not exist" Nov 01 22:24:42.085119 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.085101 5011 scope.go:122] "RemoveContainer" containerID="93e0505dc7ebcad2d5e61ba5ed9d5c84fd4a491847687ecd02182316d836ecba" Nov 01 22:24:42.085373 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:24:42.085351 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93e0505dc7ebcad2d5e61ba5ed9d5c84fd4a491847687ecd02182316d836ecba\": container with ID starting with 93e0505dc7ebcad2d5e61ba5ed9d5c84fd4a491847687ecd02182316d836ecba not found: ID does not exist" containerID="93e0505dc7ebcad2d5e61ba5ed9d5c84fd4a491847687ecd02182316d836ecba" Nov 01 22:24:42.085429 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.085374 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93e0505dc7ebcad2d5e61ba5ed9d5c84fd4a491847687ecd02182316d836ecba"} err="failed to get container status \"93e0505dc7ebcad2d5e61ba5ed9d5c84fd4a491847687ecd02182316d836ecba\": rpc error: code = NotFound desc = could not find container \"93e0505dc7ebcad2d5e61ba5ed9d5c84fd4a491847687ecd02182316d836ecba\": container with ID starting with 93e0505dc7ebcad2d5e61ba5ed9d5c84fd4a491847687ecd02182316d836ecba not found: ID does not exist" Nov 01 22:24:42.085429 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.085389 5011 scope.go:122] "RemoveContainer" containerID="6bc3a0740c090bbc8a3983f168e06b3cf7a0696dd1f9c76aae0c662e74f282c5" Nov 01 22:24:42.085620 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:24:42.085604 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bc3a0740c090bbc8a3983f168e06b3cf7a0696dd1f9c76aae0c662e74f282c5\": container with ID starting with 6bc3a0740c090bbc8a3983f168e06b3cf7a0696dd1f9c76aae0c662e74f282c5 not found: ID does not exist" containerID="6bc3a0740c090bbc8a3983f168e06b3cf7a0696dd1f9c76aae0c662e74f282c5" Nov 01 22:24:42.085658 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.085624 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bc3a0740c090bbc8a3983f168e06b3cf7a0696dd1f9c76aae0c662e74f282c5"} err="failed to get container status \"6bc3a0740c090bbc8a3983f168e06b3cf7a0696dd1f9c76aae0c662e74f282c5\": rpc error: code = NotFound desc = could not find container \"6bc3a0740c090bbc8a3983f168e06b3cf7a0696dd1f9c76aae0c662e74f282c5\": container with ID starting with 6bc3a0740c090bbc8a3983f168e06b3cf7a0696dd1f9c76aae0c662e74f282c5 not found: ID does not exist" Nov 01 22:24:42.085658 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.085637 5011 scope.go:122] "RemoveContainer" containerID="8873b887bb93503b0e9ac0ca4e5d30e30ef27bfc0a1844a03d250cfbdba3f5e9" Nov 01 22:24:42.085826 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:24:42.085807 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8873b887bb93503b0e9ac0ca4e5d30e30ef27bfc0a1844a03d250cfbdba3f5e9\": container with ID starting with 8873b887bb93503b0e9ac0ca4e5d30e30ef27bfc0a1844a03d250cfbdba3f5e9 not found: ID does not exist" containerID="8873b887bb93503b0e9ac0ca4e5d30e30ef27bfc0a1844a03d250cfbdba3f5e9" Nov 01 22:24:42.085888 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.085847 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8873b887bb93503b0e9ac0ca4e5d30e30ef27bfc0a1844a03d250cfbdba3f5e9"} err="failed to get container status \"8873b887bb93503b0e9ac0ca4e5d30e30ef27bfc0a1844a03d250cfbdba3f5e9\": rpc error: code = NotFound desc = could not find container \"8873b887bb93503b0e9ac0ca4e5d30e30ef27bfc0a1844a03d250cfbdba3f5e9\": container with ID starting with 8873b887bb93503b0e9ac0ca4e5d30e30ef27bfc0a1844a03d250cfbdba3f5e9 not found: ID does not exist" Nov 01 22:24:42.085888 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.085865 5011 scope.go:122] "RemoveContainer" containerID="acac8477146c8e4ec5bddf7d1684ff493357d1a02d26748cd5540211f41a5a51" Nov 01 22:24:42.086068 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:24:42.086052 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acac8477146c8e4ec5bddf7d1684ff493357d1a02d26748cd5540211f41a5a51\": container with ID starting with acac8477146c8e4ec5bddf7d1684ff493357d1a02d26748cd5540211f41a5a51 not found: ID does not exist" containerID="acac8477146c8e4ec5bddf7d1684ff493357d1a02d26748cd5540211f41a5a51" Nov 01 22:24:42.086102 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.086075 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acac8477146c8e4ec5bddf7d1684ff493357d1a02d26748cd5540211f41a5a51"} err="failed to get container status \"acac8477146c8e4ec5bddf7d1684ff493357d1a02d26748cd5540211f41a5a51\": rpc error: code = NotFound desc = could not find container \"acac8477146c8e4ec5bddf7d1684ff493357d1a02d26748cd5540211f41a5a51\": container with ID starting with acac8477146c8e4ec5bddf7d1684ff493357d1a02d26748cd5540211f41a5a51 not found: ID does not exist" Nov 01 22:24:42.086102 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.086089 5011 scope.go:122] "RemoveContainer" containerID="a7223bd264b3d6e9b9360c895ce6d4adb5d4d9ea532ae55d7892b6c2896c00d2" Nov 01 22:24:42.086709 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:24:42.086694 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7223bd264b3d6e9b9360c895ce6d4adb5d4d9ea532ae55d7892b6c2896c00d2\": container with ID starting with a7223bd264b3d6e9b9360c895ce6d4adb5d4d9ea532ae55d7892b6c2896c00d2 not found: ID does not exist" containerID="a7223bd264b3d6e9b9360c895ce6d4adb5d4d9ea532ae55d7892b6c2896c00d2" Nov 01 22:24:42.086761 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.086713 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7223bd264b3d6e9b9360c895ce6d4adb5d4d9ea532ae55d7892b6c2896c00d2"} err="failed to get container status \"a7223bd264b3d6e9b9360c895ce6d4adb5d4d9ea532ae55d7892b6c2896c00d2\": rpc error: code = NotFound desc = could not find container \"a7223bd264b3d6e9b9360c895ce6d4adb5d4d9ea532ae55d7892b6c2896c00d2\": container with ID starting with a7223bd264b3d6e9b9360c895ce6d4adb5d4d9ea532ae55d7892b6c2896c00d2 not found: ID does not exist" Nov 01 22:24:42.086761 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.086724 5011 scope.go:122] "RemoveContainer" containerID="d331363f4284d8fd77ab657eafb70be3c11dfa2ad3b3e96a48a46020fbc2b126" Nov 01 22:24:42.086866 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.086785 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.087202 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:24:42.086983 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d331363f4284d8fd77ab657eafb70be3c11dfa2ad3b3e96a48a46020fbc2b126\": container with ID starting with d331363f4284d8fd77ab657eafb70be3c11dfa2ad3b3e96a48a46020fbc2b126 not found: ID does not exist" containerID="d331363f4284d8fd77ab657eafb70be3c11dfa2ad3b3e96a48a46020fbc2b126" Nov 01 22:24:42.087202 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.087013 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d331363f4284d8fd77ab657eafb70be3c11dfa2ad3b3e96a48a46020fbc2b126"} err="failed to get container status \"d331363f4284d8fd77ab657eafb70be3c11dfa2ad3b3e96a48a46020fbc2b126\": rpc error: code = NotFound desc = could not find container \"d331363f4284d8fd77ab657eafb70be3c11dfa2ad3b3e96a48a46020fbc2b126\": container with ID starting with d331363f4284d8fd77ab657eafb70be3c11dfa2ad3b3e96a48a46020fbc2b126 not found: ID does not exist" Nov 01 22:24:42.090758 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.090727 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 01 22:24:42.090758 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.090732 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 01 22:24:42.090915 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.090797 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 01 22:24:42.090915 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.090806 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 01 22:24:42.091170 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.091156 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 01 22:24:42.091221 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.091166 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 01 22:24:42.091221 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.091158 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 01 22:24:42.094117 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.094100 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 01 22:24:42.094190 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.094114 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zddmv\"" Nov 01 22:24:42.094245 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.094194 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 01 22:24:42.098944 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.098910 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 22:24:42.143489 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.143457 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a5e2273-88cc-4c23-a168-c5039e5b7791-config-out\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.143489 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.143493 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a5e2273-88cc-4c23-a168-c5039e5b7791-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.143703 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.143521 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a5e2273-88cc-4c23-a168-c5039e5b7791-web-config\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.143703 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.143599 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a5e2273-88cc-4c23-a168-c5039e5b7791-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.143703 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.143617 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a5e2273-88cc-4c23-a168-c5039e5b7791-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.143703 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.143684 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5a5e2273-88cc-4c23-a168-c5039e5b7791-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.143874 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.143720 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5a5e2273-88cc-4c23-a168-c5039e5b7791-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.143874 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.143737 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzc9d\" (UniqueName: \"kubernetes.io/projected/5a5e2273-88cc-4c23-a168-c5039e5b7791-kube-api-access-lzc9d\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.143874 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.143755 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a5e2273-88cc-4c23-a168-c5039e5b7791-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.143874 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.143812 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a5e2273-88cc-4c23-a168-c5039e5b7791-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.143995 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.143886 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5a5e2273-88cc-4c23-a168-c5039e5b7791-config-volume\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.143995 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.143933 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5a5e2273-88cc-4c23-a168-c5039e5b7791-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.143995 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.143953 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5a5e2273-88cc-4c23-a168-c5039e5b7791-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.245061 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.245013 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5a5e2273-88cc-4c23-a168-c5039e5b7791-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.245061 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.245060 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzc9d\" (UniqueName: \"kubernetes.io/projected/5a5e2273-88cc-4c23-a168-c5039e5b7791-kube-api-access-lzc9d\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.245314 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.245085 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a5e2273-88cc-4c23-a168-c5039e5b7791-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.245314 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.245115 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a5e2273-88cc-4c23-a168-c5039e5b7791-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.245314 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.245295 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5a5e2273-88cc-4c23-a168-c5039e5b7791-config-volume\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.245478 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.245343 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5a5e2273-88cc-4c23-a168-c5039e5b7791-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.245478 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.245370 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5a5e2273-88cc-4c23-a168-c5039e5b7791-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.245478 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.245419 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a5e2273-88cc-4c23-a168-c5039e5b7791-config-out\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.245478 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.245452 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a5e2273-88cc-4c23-a168-c5039e5b7791-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.245478 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.245470 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5a5e2273-88cc-4c23-a168-c5039e5b7791-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.245735 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.245486 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a5e2273-88cc-4c23-a168-c5039e5b7791-web-config\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.245735 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.245527 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a5e2273-88cc-4c23-a168-c5039e5b7791-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.245735 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.245552 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a5e2273-88cc-4c23-a168-c5039e5b7791-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.245735 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.245588 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5a5e2273-88cc-4c23-a168-c5039e5b7791-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.246049 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.246027 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a5e2273-88cc-4c23-a168-c5039e5b7791-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.248709 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.248591 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a5e2273-88cc-4c23-a168-c5039e5b7791-config-out\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.248709 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.248697 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a5e2273-88cc-4c23-a168-c5039e5b7791-web-config\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.248709 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.248704 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5a5e2273-88cc-4c23-a168-c5039e5b7791-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.248938 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.248709 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5a5e2273-88cc-4c23-a168-c5039e5b7791-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.248938 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.248748 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5a5e2273-88cc-4c23-a168-c5039e5b7791-config-volume\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.248938 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.248868 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a5e2273-88cc-4c23-a168-c5039e5b7791-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.249105 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.249083 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a5e2273-88cc-4c23-a168-c5039e5b7791-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.249149 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.249129 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a5e2273-88cc-4c23-a168-c5039e5b7791-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.249220 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.249206 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5a5e2273-88cc-4c23-a168-c5039e5b7791-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.250667 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.250651 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a5e2273-88cc-4c23-a168-c5039e5b7791-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.253007 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.252986 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzc9d\" (UniqueName: \"kubernetes.io/projected/5a5e2273-88cc-4c23-a168-c5039e5b7791-kube-api-access-lzc9d\") pod \"alertmanager-main-0\" (UID: \"5a5e2273-88cc-4c23-a168-c5039e5b7791\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.413870 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.413751 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 22:24:42.694487 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.694454 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 22:24:42.698984 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:24:42.698948 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a5e2273_88cc_4c23_a168_c5039e5b7791.slice/crio-944dd927155d2b758f77f3507a4e8c800f963a67849e33e68237b0bc4e35c3d5 WatchSource:0}: Error finding container 944dd927155d2b758f77f3507a4e8c800f963a67849e33e68237b0bc4e35c3d5: Status 404 returned error can't find the container with id 944dd927155d2b758f77f3507a4e8c800f963a67849e33e68237b0bc4e35c3d5 Nov 01 22:24:42.873315 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.873283 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-c9c7cf75c-4qs78"] Nov 01 22:24:42.876286 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.876270 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c9c7cf75c-4qs78" Nov 01 22:24:42.886648 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.886594 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c9c7cf75c-4qs78"] Nov 01 22:24:42.950517 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.950423 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf1d3e40-6dcb-490f-82c4-ac196ff2589d-oauth-serving-cert\") pod \"console-c9c7cf75c-4qs78\" (UID: \"bf1d3e40-6dcb-490f-82c4-ac196ff2589d\") " pod="openshift-console/console-c9c7cf75c-4qs78" Nov 01 22:24:42.950517 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.950463 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf1d3e40-6dcb-490f-82c4-ac196ff2589d-tmp\") pod \"console-c9c7cf75c-4qs78\" (UID: \"bf1d3e40-6dcb-490f-82c4-ac196ff2589d\") " pod="openshift-console/console-c9c7cf75c-4qs78" Nov 01 22:24:42.950517 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.950488 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf1d3e40-6dcb-490f-82c4-ac196ff2589d-console-serving-cert\") pod \"console-c9c7cf75c-4qs78\" (UID: \"bf1d3e40-6dcb-490f-82c4-ac196ff2589d\") " pod="openshift-console/console-c9c7cf75c-4qs78" Nov 01 22:24:42.950874 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.950613 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf1d3e40-6dcb-490f-82c4-ac196ff2589d-service-ca\") pod \"console-c9c7cf75c-4qs78\" (UID: \"bf1d3e40-6dcb-490f-82c4-ac196ff2589d\") " pod="openshift-console/console-c9c7cf75c-4qs78" Nov 01 22:24:42.950874 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.950693 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf1d3e40-6dcb-490f-82c4-ac196ff2589d-console-oauth-config\") pod \"console-c9c7cf75c-4qs78\" (UID: \"bf1d3e40-6dcb-490f-82c4-ac196ff2589d\") " pod="openshift-console/console-c9c7cf75c-4qs78" Nov 01 22:24:42.950874 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.950730 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d27fc\" (UniqueName: \"kubernetes.io/projected/bf1d3e40-6dcb-490f-82c4-ac196ff2589d-kube-api-access-d27fc\") pod \"console-c9c7cf75c-4qs78\" (UID: \"bf1d3e40-6dcb-490f-82c4-ac196ff2589d\") " pod="openshift-console/console-c9c7cf75c-4qs78" Nov 01 22:24:42.950874 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.950786 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf1d3e40-6dcb-490f-82c4-ac196ff2589d-trusted-ca-bundle\") pod \"console-c9c7cf75c-4qs78\" (UID: \"bf1d3e40-6dcb-490f-82c4-ac196ff2589d\") " pod="openshift-console/console-c9c7cf75c-4qs78" Nov 01 22:24:42.950874 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:42.950859 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf1d3e40-6dcb-490f-82c4-ac196ff2589d-console-config\") pod \"console-c9c7cf75c-4qs78\" (UID: \"bf1d3e40-6dcb-490f-82c4-ac196ff2589d\") " pod="openshift-console/console-c9c7cf75c-4qs78" Nov 01 22:24:43.037718 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:43.037683 5011 generic.go:356] "Generic (PLEG): container finished" podID="5a5e2273-88cc-4c23-a168-c5039e5b7791" containerID="a105f4a23776c58a1b422a784bf6795cee8303a47990d982e1bff34f334f5597" exitCode=0 Nov 01 22:24:43.037882 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:43.037731 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a5e2273-88cc-4c23-a168-c5039e5b7791","Type":"ContainerDied","Data":"a105f4a23776c58a1b422a784bf6795cee8303a47990d982e1bff34f334f5597"} Nov 01 22:24:43.037882 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:43.037780 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a5e2273-88cc-4c23-a168-c5039e5b7791","Type":"ContainerStarted","Data":"944dd927155d2b758f77f3507a4e8c800f963a67849e33e68237b0bc4e35c3d5"} Nov 01 22:24:43.051877 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:43.051854 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf1d3e40-6dcb-490f-82c4-ac196ff2589d-console-config\") pod \"console-c9c7cf75c-4qs78\" (UID: \"bf1d3e40-6dcb-490f-82c4-ac196ff2589d\") " pod="openshift-console/console-c9c7cf75c-4qs78" Nov 01 22:24:43.051985 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:43.051889 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf1d3e40-6dcb-490f-82c4-ac196ff2589d-oauth-serving-cert\") pod \"console-c9c7cf75c-4qs78\" (UID: \"bf1d3e40-6dcb-490f-82c4-ac196ff2589d\") " pod="openshift-console/console-c9c7cf75c-4qs78" Nov 01 22:24:43.051985 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:43.051909 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf1d3e40-6dcb-490f-82c4-ac196ff2589d-tmp\") pod \"console-c9c7cf75c-4qs78\" (UID: \"bf1d3e40-6dcb-490f-82c4-ac196ff2589d\") " pod="openshift-console/console-c9c7cf75c-4qs78" Nov 01 22:24:43.051985 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:43.051936 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf1d3e40-6dcb-490f-82c4-ac196ff2589d-console-serving-cert\") pod \"console-c9c7cf75c-4qs78\" (UID: \"bf1d3e40-6dcb-490f-82c4-ac196ff2589d\") " pod="openshift-console/console-c9c7cf75c-4qs78" Nov 01 22:24:43.051985 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:43.051981 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf1d3e40-6dcb-490f-82c4-ac196ff2589d-service-ca\") pod \"console-c9c7cf75c-4qs78\" (UID: \"bf1d3e40-6dcb-490f-82c4-ac196ff2589d\") " pod="openshift-console/console-c9c7cf75c-4qs78" Nov 01 22:24:43.052169 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:43.052013 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf1d3e40-6dcb-490f-82c4-ac196ff2589d-console-oauth-config\") pod \"console-c9c7cf75c-4qs78\" (UID: \"bf1d3e40-6dcb-490f-82c4-ac196ff2589d\") " pod="openshift-console/console-c9c7cf75c-4qs78" Nov 01 22:24:43.052169 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:43.052046 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d27fc\" (UniqueName: \"kubernetes.io/projected/bf1d3e40-6dcb-490f-82c4-ac196ff2589d-kube-api-access-d27fc\") pod \"console-c9c7cf75c-4qs78\" (UID: \"bf1d3e40-6dcb-490f-82c4-ac196ff2589d\") " pod="openshift-console/console-c9c7cf75c-4qs78" Nov 01 22:24:43.052169 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:43.052073 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf1d3e40-6dcb-490f-82c4-ac196ff2589d-trusted-ca-bundle\") pod \"console-c9c7cf75c-4qs78\" (UID: \"bf1d3e40-6dcb-490f-82c4-ac196ff2589d\") " pod="openshift-console/console-c9c7cf75c-4qs78" Nov 01 22:24:43.052915 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:43.052487 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf1d3e40-6dcb-490f-82c4-ac196ff2589d-tmp\") pod \"console-c9c7cf75c-4qs78\" (UID: \"bf1d3e40-6dcb-490f-82c4-ac196ff2589d\") " pod="openshift-console/console-c9c7cf75c-4qs78" Nov 01 22:24:43.052915 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:43.052583 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf1d3e40-6dcb-490f-82c4-ac196ff2589d-console-config\") pod \"console-c9c7cf75c-4qs78\" (UID: \"bf1d3e40-6dcb-490f-82c4-ac196ff2589d\") " pod="openshift-console/console-c9c7cf75c-4qs78" Nov 01 22:24:43.052915 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:43.052757 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf1d3e40-6dcb-490f-82c4-ac196ff2589d-oauth-serving-cert\") pod \"console-c9c7cf75c-4qs78\" (UID: \"bf1d3e40-6dcb-490f-82c4-ac196ff2589d\") " pod="openshift-console/console-c9c7cf75c-4qs78" Nov 01 22:24:43.053298 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:43.052928 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf1d3e40-6dcb-490f-82c4-ac196ff2589d-service-ca\") pod \"console-c9c7cf75c-4qs78\" (UID: \"bf1d3e40-6dcb-490f-82c4-ac196ff2589d\") " pod="openshift-console/console-c9c7cf75c-4qs78" Nov 01 22:24:43.053298 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:43.053099 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf1d3e40-6dcb-490f-82c4-ac196ff2589d-trusted-ca-bundle\") pod \"console-c9c7cf75c-4qs78\" (UID: \"bf1d3e40-6dcb-490f-82c4-ac196ff2589d\") " pod="openshift-console/console-c9c7cf75c-4qs78" Nov 01 22:24:43.054341 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:43.054322 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf1d3e40-6dcb-490f-82c4-ac196ff2589d-console-serving-cert\") pod \"console-c9c7cf75c-4qs78\" (UID: \"bf1d3e40-6dcb-490f-82c4-ac196ff2589d\") " pod="openshift-console/console-c9c7cf75c-4qs78" Nov 01 22:24:43.054998 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:43.054980 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf1d3e40-6dcb-490f-82c4-ac196ff2589d-console-oauth-config\") pod \"console-c9c7cf75c-4qs78\" (UID: \"bf1d3e40-6dcb-490f-82c4-ac196ff2589d\") " pod="openshift-console/console-c9c7cf75c-4qs78" Nov 01 22:24:43.060399 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:43.060367 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d27fc\" (UniqueName: \"kubernetes.io/projected/bf1d3e40-6dcb-490f-82c4-ac196ff2589d-kube-api-access-d27fc\") pod \"console-c9c7cf75c-4qs78\" (UID: \"bf1d3e40-6dcb-490f-82c4-ac196ff2589d\") " pod="openshift-console/console-c9c7cf75c-4qs78" Nov 01 22:24:43.187849 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:43.187809 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c9c7cf75c-4qs78" Nov 01 22:24:43.463633 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:43.463562 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c9c7cf75c-4qs78"] Nov 01 22:24:43.467531 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:24:43.467505 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf1d3e40_6dcb_490f_82c4_ac196ff2589d.slice/crio-d884ea7a4481ddcfdb5a4822c86250937d13c93aa9b7e29f0af0160acf2fc82d WatchSource:0}: Error finding container d884ea7a4481ddcfdb5a4822c86250937d13c93aa9b7e29f0af0160acf2fc82d: Status 404 returned error can't find the container with id d884ea7a4481ddcfdb5a4822c86250937d13c93aa9b7e29f0af0160acf2fc82d Nov 01 22:24:43.744474 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:43.744399 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5349ab21-2645-4464-a07c-d71e4503808d" path="/var/lib/kubelet/pods/5349ab21-2645-4464-a07c-d71e4503808d/volumes" Nov 01 22:24:44.044074 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:44.043964 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c9c7cf75c-4qs78" event={"ID":"bf1d3e40-6dcb-490f-82c4-ac196ff2589d","Type":"ContainerStarted","Data":"6253df6749e7f6deedba96f6afa6f1ba0224fa27045e896e6cba1461eb1f2116"} Nov 01 22:24:44.044074 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:44.044030 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c9c7cf75c-4qs78" event={"ID":"bf1d3e40-6dcb-490f-82c4-ac196ff2589d","Type":"ContainerStarted","Data":"d884ea7a4481ddcfdb5a4822c86250937d13c93aa9b7e29f0af0160acf2fc82d"} Nov 01 22:24:44.047261 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:44.047238 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a5e2273-88cc-4c23-a168-c5039e5b7791","Type":"ContainerStarted","Data":"af4abb161759c8cb090cf227127a303d45791c67c477fa2d439c9bb6b04acefd"} Nov 01 22:24:44.047387 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:44.047286 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a5e2273-88cc-4c23-a168-c5039e5b7791","Type":"ContainerStarted","Data":"e54be68d088e207a02668dce5c3a0babd3d5c62880ef37bb923ee91d9dd0cbbe"} Nov 01 22:24:44.047387 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:44.047313 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a5e2273-88cc-4c23-a168-c5039e5b7791","Type":"ContainerStarted","Data":"34ab600941f18fca5221e2d13c74f7e942a72313866ebbf2480a4bb678df194f"} Nov 01 22:24:44.047387 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:44.047337 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a5e2273-88cc-4c23-a168-c5039e5b7791","Type":"ContainerStarted","Data":"4561fdda83d29863db8571e823b3bb8ee485637a17fc46bf21596c536d4b04fa"} Nov 01 22:24:44.047387 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:44.047365 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a5e2273-88cc-4c23-a168-c5039e5b7791","Type":"ContainerStarted","Data":"0e5e124083cb14d3f44bba7644a35030c50937384211314749c81f7bf88587d1"} Nov 01 22:24:44.047387 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:44.047388 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a5e2273-88cc-4c23-a168-c5039e5b7791","Type":"ContainerStarted","Data":"7da8042a2f6b82ef558d524c37864c1e000a0d8757dca3c480ddd6b60a888318"} Nov 01 22:24:44.061750 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:44.061704 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-c9c7cf75c-4qs78" podStartSLOduration=2.061690437 podStartE2EDuration="2.061690437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:44.061254811 +0000 UTC m=+52.465845443" watchObservedRunningTime="2026-11-01 22:24:44.061690437 +0000 UTC m=+52.466281138" Nov 01 22:24:44.087712 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:44.087657 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.074551289 podStartE2EDuration="2.087631087s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.079798ms" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:44.085379019 +0000 UTC m=+52.489969670" watchObservedRunningTime="2026-11-01 22:24:44.087631087 +0000 UTC m=+52.492221720" Nov 01 22:24:46.958497 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:24:46.958457 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:24:53.188707 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:53.188658 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c9c7cf75c-4qs78" Nov 01 22:24:53.189160 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:53.188774 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c9c7cf75c-4qs78" Nov 01 22:24:53.194361 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:53.194340 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c9c7cf75c-4qs78" Nov 01 22:24:54.087614 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:54.087584 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c9c7cf75c-4qs78" Nov 01 22:24:54.134034 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:24:54.134001 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fd74c69cd-dlgmp"] Nov 01 22:25:16.958243 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:25:16.958206 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:25:19.156570 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:19.156507 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5fd74c69cd-dlgmp" podUID="248bc413-99de-4b80-a73e-6e342fb22587" containerName="console" containerID="cri-o://13359b44df6cf101cbe21cebc18b6c7066150d1b1c8d676a8e325df4018dd6e9" gracePeriod=15 Nov 01 22:25:19.393787 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:19.393763 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fd74c69cd-dlgmp" Nov 01 22:25:19.455267 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:19.455233 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/248bc413-99de-4b80-a73e-6e342fb22587-service-ca\" (UniqueName: \"kubernetes.io/configmap/248bc413-99de-4b80-a73e-6e342fb22587-service-ca\") pod \"248bc413-99de-4b80-a73e-6e342fb22587\" (UID: \"248bc413-99de-4b80-a73e-6e342fb22587\") " Nov 01 22:25:19.455435 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:19.455275 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/248bc413-99de-4b80-a73e-6e342fb22587-kube-api-access-nq8pw\" (UniqueName: \"kubernetes.io/projected/248bc413-99de-4b80-a73e-6e342fb22587-kube-api-access-nq8pw\") pod \"248bc413-99de-4b80-a73e-6e342fb22587\" (UID: \"248bc413-99de-4b80-a73e-6e342fb22587\") " Nov 01 22:25:19.455435 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:19.455303 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/248bc413-99de-4b80-a73e-6e342fb22587-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/248bc413-99de-4b80-a73e-6e342fb22587-trusted-ca-bundle\") pod \"248bc413-99de-4b80-a73e-6e342fb22587\" (UID: \"248bc413-99de-4b80-a73e-6e342fb22587\") " Nov 01 22:25:19.455629 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:19.455425 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/248bc413-99de-4b80-a73e-6e342fb22587-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/248bc413-99de-4b80-a73e-6e342fb22587-console-oauth-config\") pod \"248bc413-99de-4b80-a73e-6e342fb22587\" (UID: \"248bc413-99de-4b80-a73e-6e342fb22587\") " Nov 01 22:25:19.455629 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:19.455469 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/248bc413-99de-4b80-a73e-6e342fb22587-tmp\" (UniqueName: \"kubernetes.io/empty-dir/248bc413-99de-4b80-a73e-6e342fb22587-tmp\") pod \"248bc413-99de-4b80-a73e-6e342fb22587\" (UID: \"248bc413-99de-4b80-a73e-6e342fb22587\") " Nov 01 22:25:19.455629 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:19.455505 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/248bc413-99de-4b80-a73e-6e342fb22587-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/248bc413-99de-4b80-a73e-6e342fb22587-console-serving-cert\") pod \"248bc413-99de-4b80-a73e-6e342fb22587\" (UID: \"248bc413-99de-4b80-a73e-6e342fb22587\") " Nov 01 22:25:19.455629 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:19.455541 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/248bc413-99de-4b80-a73e-6e342fb22587-console-config\" (UniqueName: \"kubernetes.io/configmap/248bc413-99de-4b80-a73e-6e342fb22587-console-config\") pod \"248bc413-99de-4b80-a73e-6e342fb22587\" (UID: \"248bc413-99de-4b80-a73e-6e342fb22587\") " Nov 01 22:25:19.455629 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:19.455569 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/248bc413-99de-4b80-a73e-6e342fb22587-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/248bc413-99de-4b80-a73e-6e342fb22587-oauth-serving-cert\") pod \"248bc413-99de-4b80-a73e-6e342fb22587\" (UID: \"248bc413-99de-4b80-a73e-6e342fb22587\") " Nov 01 22:25:19.455914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:19.455689 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/248bc413-99de-4b80-a73e-6e342fb22587-trusted-ca-bundle" pod "248bc413-99de-4b80-a73e-6e342fb22587" (UID: "248bc413-99de-4b80-a73e-6e342fb22587"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:25:19.455914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:19.455694 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/248bc413-99de-4b80-a73e-6e342fb22587-service-ca" pod "248bc413-99de-4b80-a73e-6e342fb22587" (UID: "248bc413-99de-4b80-a73e-6e342fb22587"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:25:19.455914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:19.455793 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/248bc413-99de-4b80-a73e-6e342fb22587-tmp" pod "248bc413-99de-4b80-a73e-6e342fb22587" (UID: "248bc413-99de-4b80-a73e-6e342fb22587"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:25:19.455914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:19.455888 5011 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/248bc413-99de-4b80-a73e-6e342fb22587-trusted-ca-bundle\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:19.456171 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:19.455984 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/248bc413-99de-4b80-a73e-6e342fb22587-console-config" pod "248bc413-99de-4b80-a73e-6e342fb22587" (UID: "248bc413-99de-4b80-a73e-6e342fb22587"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:25:19.456171 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:19.456066 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/248bc413-99de-4b80-a73e-6e342fb22587-oauth-serving-cert" pod "248bc413-99de-4b80-a73e-6e342fb22587" (UID: "248bc413-99de-4b80-a73e-6e342fb22587"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:25:19.457702 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:19.457679 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/248bc413-99de-4b80-a73e-6e342fb22587-console-oauth-config" pod "248bc413-99de-4b80-a73e-6e342fb22587" (UID: "248bc413-99de-4b80-a73e-6e342fb22587"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:25:19.457702 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:19.457678 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/248bc413-99de-4b80-a73e-6e342fb22587-kube-api-access-nq8pw" pod "248bc413-99de-4b80-a73e-6e342fb22587" (UID: "248bc413-99de-4b80-a73e-6e342fb22587"). InnerVolumeSpecName "kube-api-access-nq8pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:25:19.457970 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:19.457954 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/248bc413-99de-4b80-a73e-6e342fb22587-console-serving-cert" pod "248bc413-99de-4b80-a73e-6e342fb22587" (UID: "248bc413-99de-4b80-a73e-6e342fb22587"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:25:19.556482 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:19.556447 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq8pw\" (UniqueName: \"kubernetes.io/projected/248bc413-99de-4b80-a73e-6e342fb22587-kube-api-access-nq8pw\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:19.556482 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:19.556477 5011 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/248bc413-99de-4b80-a73e-6e342fb22587-console-oauth-config\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:19.556482 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:19.556488 5011 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/248bc413-99de-4b80-a73e-6e342fb22587-tmp\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:19.556710 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:19.556498 5011 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/248bc413-99de-4b80-a73e-6e342fb22587-console-serving-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:19.556710 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:19.556508 5011 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/248bc413-99de-4b80-a73e-6e342fb22587-console-config\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:19.556710 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:19.556516 5011 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/248bc413-99de-4b80-a73e-6e342fb22587-oauth-serving-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:19.556710 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:19.556524 5011 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/248bc413-99de-4b80-a73e-6e342fb22587-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:20.171904 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:20.171868 5011 generic.go:356] "Generic (PLEG): container finished" podID="248bc413-99de-4b80-a73e-6e342fb22587" containerID="13359b44df6cf101cbe21cebc18b6c7066150d1b1c8d676a8e325df4018dd6e9" exitCode=0 Nov 01 22:25:20.172366 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:20.171923 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fd74c69cd-dlgmp" Nov 01 22:25:20.172366 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:20.171940 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fd74c69cd-dlgmp" event={"ID":"248bc413-99de-4b80-a73e-6e342fb22587","Type":"ContainerDied","Data":"13359b44df6cf101cbe21cebc18b6c7066150d1b1c8d676a8e325df4018dd6e9"} Nov 01 22:25:20.172366 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:20.171996 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fd74c69cd-dlgmp" event={"ID":"248bc413-99de-4b80-a73e-6e342fb22587","Type":"ContainerDied","Data":"cae9ff1fcf85a91cb7c03f9f1a8dafa0170fe28900a52ca314eb3872588b0483"} Nov 01 22:25:20.172366 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:20.172025 5011 scope.go:122] "RemoveContainer" containerID="13359b44df6cf101cbe21cebc18b6c7066150d1b1c8d676a8e325df4018dd6e9" Nov 01 22:25:20.180071 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:20.180055 5011 scope.go:122] "RemoveContainer" containerID="13359b44df6cf101cbe21cebc18b6c7066150d1b1c8d676a8e325df4018dd6e9" Nov 01 22:25:20.180373 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:25:20.180345 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13359b44df6cf101cbe21cebc18b6c7066150d1b1c8d676a8e325df4018dd6e9\": container with ID starting with 13359b44df6cf101cbe21cebc18b6c7066150d1b1c8d676a8e325df4018dd6e9 not found: ID does not exist" containerID="13359b44df6cf101cbe21cebc18b6c7066150d1b1c8d676a8e325df4018dd6e9" Nov 01 22:25:20.180419 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:20.180383 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13359b44df6cf101cbe21cebc18b6c7066150d1b1c8d676a8e325df4018dd6e9"} err="failed to get container status \"13359b44df6cf101cbe21cebc18b6c7066150d1b1c8d676a8e325df4018dd6e9\": rpc error: code = NotFound desc = could not find container \"13359b44df6cf101cbe21cebc18b6c7066150d1b1c8d676a8e325df4018dd6e9\": container with ID starting with 13359b44df6cf101cbe21cebc18b6c7066150d1b1c8d676a8e325df4018dd6e9 not found: ID does not exist" Nov 01 22:25:20.189038 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:20.189017 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fd74c69cd-dlgmp"] Nov 01 22:25:20.192808 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:20.192789 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5fd74c69cd-dlgmp"] Nov 01 22:25:21.743626 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:21.743596 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="248bc413-99de-4b80-a73e-6e342fb22587" path="/var/lib/kubelet/pods/248bc413-99de-4b80-a73e-6e342fb22587/volumes" Nov 01 22:25:26.194188 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:26.194152 5011 generic.go:356] "Generic (PLEG): container finished" podID="ed5e5b02-53ae-4da1-9ed5-43815cb00ce0" containerID="6ffa13f23e352c981e985300b4317eafa0a656f0533206363a7e3d3a48d9c8e0" exitCode=0 Nov 01 22:25:26.194602 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:26.194211 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hpj5f-7thv7" event={"ID":"ed5e5b02-53ae-4da1-9ed5-43815cb00ce0","Type":"ContainerDied","Data":"6ffa13f23e352c981e985300b4317eafa0a656f0533206363a7e3d3a48d9c8e0"} Nov 01 22:25:27.328343 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:27.328317 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hpj5f-7thv7" Nov 01 22:25:27.406773 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:27.406731 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed5e5b02-53ae-4da1-9ed5-43815cb00ce0-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed5e5b02-53ae-4da1-9ed5-43815cb00ce0-trusted-ca-bundle\") pod \"ed5e5b02-53ae-4da1-9ed5-43815cb00ce0\" (UID: \"ed5e5b02-53ae-4da1-9ed5-43815cb00ce0\") " Nov 01 22:25:27.406986 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:27.406797 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed5e5b02-53ae-4da1-9ed5-43815cb00ce0-kube-api-access-pxs2f\" (UniqueName: \"kubernetes.io/projected/ed5e5b02-53ae-4da1-9ed5-43815cb00ce0-kube-api-access-pxs2f\") pod \"ed5e5b02-53ae-4da1-9ed5-43815cb00ce0\" (UID: \"ed5e5b02-53ae-4da1-9ed5-43815cb00ce0\") " Nov 01 22:25:27.406986 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:27.406823 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed5e5b02-53ae-4da1-9ed5-43815cb00ce0-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed5e5b02-53ae-4da1-9ed5-43815cb00ce0-service-ca-bundle\") pod \"ed5e5b02-53ae-4da1-9ed5-43815cb00ce0\" (UID: \"ed5e5b02-53ae-4da1-9ed5-43815cb00ce0\") " Nov 01 22:25:27.407192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:27.407167 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed5e5b02-53ae-4da1-9ed5-43815cb00ce0-trusted-ca-bundle" pod "ed5e5b02-53ae-4da1-9ed5-43815cb00ce0" (UID: "ed5e5b02-53ae-4da1-9ed5-43815cb00ce0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:25:27.407246 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:27.407189 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed5e5b02-53ae-4da1-9ed5-43815cb00ce0-service-ca-bundle" pod "ed5e5b02-53ae-4da1-9ed5-43815cb00ce0" (UID: "ed5e5b02-53ae-4da1-9ed5-43815cb00ce0"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:25:27.409080 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:27.409055 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed5e5b02-53ae-4da1-9ed5-43815cb00ce0-kube-api-access-pxs2f" pod "ed5e5b02-53ae-4da1-9ed5-43815cb00ce0" (UID: "ed5e5b02-53ae-4da1-9ed5-43815cb00ce0"). InnerVolumeSpecName "kube-api-access-pxs2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:25:27.508114 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:27.508030 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed5e5b02-53ae-4da1-9ed5-43815cb00ce0-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ed5e5b02-53ae-4da1-9ed5-43815cb00ce0-archives-path\") pod \"ed5e5b02-53ae-4da1-9ed5-43815cb00ce0\" (UID: \"ed5e5b02-53ae-4da1-9ed5-43815cb00ce0\") " Nov 01 22:25:27.508255 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:27.508197 5011 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed5e5b02-53ae-4da1-9ed5-43815cb00ce0-trusted-ca-bundle\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:27.508255 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:27.508209 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxs2f\" (UniqueName: \"kubernetes.io/projected/ed5e5b02-53ae-4da1-9ed5-43815cb00ce0-kube-api-access-pxs2f\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:27.508255 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:27.508221 5011 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed5e5b02-53ae-4da1-9ed5-43815cb00ce0-service-ca-bundle\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:27.508350 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:27.508331 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5e5b02-53ae-4da1-9ed5-43815cb00ce0-archives-path" pod "ed5e5b02-53ae-4da1-9ed5-43815cb00ce0" (UID: "ed5e5b02-53ae-4da1-9ed5-43815cb00ce0"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:25:27.608567 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:27.608529 5011 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ed5e5b02-53ae-4da1-9ed5-43815cb00ce0-archives-path\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:25:28.202145 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:28.202108 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hpj5f-7thv7" event={"ID":"ed5e5b02-53ae-4da1-9ed5-43815cb00ce0","Type":"ContainerDied","Data":"d91f5adb2d727f28228ccaf880a918818cd5d1ef1a3bb199834d6806e66a351f"} Nov 01 22:25:28.202321 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:28.202158 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d91f5adb2d727f28228ccaf880a918818cd5d1ef1a3bb199834d6806e66a351f" Nov 01 22:25:28.202321 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:25:28.202118 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hpj5f-7thv7" Nov 01 22:25:46.957611 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:25:46.957532 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:26:16.957433 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:26:16.957389 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:26:46.961020 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:26:46.960981 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:27:16.959445 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:27:16.959360 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:27:46.958019 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:27:46.957979 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:28:16.958598 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:28:16.958556 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:28:46.959402 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:28:46.959316 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:29:09.781886 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:09.781849 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qkpgg"] Nov 01 22:29:09.782410 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:09.782112 5011 cpu_manager.go:414] "containerMap: removing container" podUID="248bc413-99de-4b80-a73e-6e342fb22587" containerName="console" Nov 01 22:29:09.782410 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:09.782122 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="248bc413-99de-4b80-a73e-6e342fb22587" containerName="console" Nov 01 22:29:09.782410 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:09.782139 5011 cpu_manager.go:414] "containerMap: removing container" podUID="ed5e5b02-53ae-4da1-9ed5-43815cb00ce0" containerName="insights-gathering" Nov 01 22:29:09.782410 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:09.782145 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed5e5b02-53ae-4da1-9ed5-43815cb00ce0" containerName="insights-gathering" Nov 01 22:29:09.782410 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:09.782189 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed5e5b02-53ae-4da1-9ed5-43815cb00ce0" containerName="insights-gathering" Nov 01 22:29:09.782410 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:09.782196 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="248bc413-99de-4b80-a73e-6e342fb22587" containerName="console" Nov 01 22:29:09.785146 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:09.785129 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qkpgg" Nov 01 22:29:09.789799 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:09.789781 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 01 22:29:09.789911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:09.789798 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 01 22:29:09.813589 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:09.813565 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-7vtnw\"" Nov 01 22:29:09.813815 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:09.813795 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7afb798-1ba6-440d-9ba2-b2da734044b2-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qkpgg\" (UID: \"a7afb798-1ba6-440d-9ba2-b2da734044b2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qkpgg" Nov 01 22:29:09.813905 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:09.813850 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jfbd\" (UniqueName: \"kubernetes.io/projected/a7afb798-1ba6-440d-9ba2-b2da734044b2-kube-api-access-7jfbd\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qkpgg\" (UID: \"a7afb798-1ba6-440d-9ba2-b2da734044b2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qkpgg" Nov 01 22:29:09.814001 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:09.813979 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7afb798-1ba6-440d-9ba2-b2da734044b2-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qkpgg\" (UID: \"a7afb798-1ba6-440d-9ba2-b2da734044b2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qkpgg" Nov 01 22:29:09.838279 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:09.838254 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qkpgg"] Nov 01 22:29:09.914966 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:09.914936 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7afb798-1ba6-440d-9ba2-b2da734044b2-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qkpgg\" (UID: \"a7afb798-1ba6-440d-9ba2-b2da734044b2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qkpgg" Nov 01 22:29:09.915139 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:09.914983 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7afb798-1ba6-440d-9ba2-b2da734044b2-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qkpgg\" (UID: \"a7afb798-1ba6-440d-9ba2-b2da734044b2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qkpgg" Nov 01 22:29:09.915139 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:09.915116 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jfbd\" (UniqueName: \"kubernetes.io/projected/a7afb798-1ba6-440d-9ba2-b2da734044b2-kube-api-access-7jfbd\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qkpgg\" (UID: \"a7afb798-1ba6-440d-9ba2-b2da734044b2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qkpgg" Nov 01 22:29:09.915303 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:09.915288 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7afb798-1ba6-440d-9ba2-b2da734044b2-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qkpgg\" (UID: \"a7afb798-1ba6-440d-9ba2-b2da734044b2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qkpgg" Nov 01 22:29:09.915345 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:09.915325 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7afb798-1ba6-440d-9ba2-b2da734044b2-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qkpgg\" (UID: \"a7afb798-1ba6-440d-9ba2-b2da734044b2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qkpgg" Nov 01 22:29:09.923893 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:09.923868 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jfbd\" (UniqueName: \"kubernetes.io/projected/a7afb798-1ba6-440d-9ba2-b2da734044b2-kube-api-access-7jfbd\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qkpgg\" (UID: \"a7afb798-1ba6-440d-9ba2-b2da734044b2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qkpgg" Nov 01 22:29:09.932360 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:09.932339 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pkdpb"] Nov 01 22:29:09.935758 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:09.935743 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pkdpb" Nov 01 22:29:09.938999 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:09.938980 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-b7w4j\"" Nov 01 22:29:09.944865 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:09.944821 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pkdpb"] Nov 01 22:29:10.015909 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.015875 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddbdcba2-7288-42eb-a7ba-c6b57a338e47-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pkdpb\" (UID: \"ddbdcba2-7288-42eb-a7ba-c6b57a338e47\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pkdpb" Nov 01 22:29:10.016071 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.015921 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9wrt\" (UniqueName: \"kubernetes.io/projected/ddbdcba2-7288-42eb-a7ba-c6b57a338e47-kube-api-access-g9wrt\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pkdpb\" (UID: \"ddbdcba2-7288-42eb-a7ba-c6b57a338e47\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pkdpb" Nov 01 22:29:10.016071 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.016032 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddbdcba2-7288-42eb-a7ba-c6b57a338e47-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pkdpb\" (UID: \"ddbdcba2-7288-42eb-a7ba-c6b57a338e47\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pkdpb" Nov 01 22:29:10.078245 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.078168 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96"] Nov 01 22:29:10.081311 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.081295 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" Nov 01 22:29:10.085295 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.085275 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 01 22:29:10.085895 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.085875 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 22:29:10.085895 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.085888 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 01 22:29:10.086067 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.085948 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 01 22:29:10.086067 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.085997 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 01 22:29:10.086986 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.086967 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 01 22:29:10.087080 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.087057 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xhx7b\"" Nov 01 22:29:10.100742 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.100721 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qkpgg" Nov 01 22:29:10.103003 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.102982 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96"] Nov 01 22:29:10.116872 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.116844 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/571cb0bc-4b05-4d60-968d-44b609698f40-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-96w96\" (UID: \"571cb0bc-4b05-4d60-968d-44b609698f40\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" Nov 01 22:29:10.117024 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.116892 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddbdcba2-7288-42eb-a7ba-c6b57a338e47-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pkdpb\" (UID: \"ddbdcba2-7288-42eb-a7ba-c6b57a338e47\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pkdpb" Nov 01 22:29:10.117024 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.116919 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/571cb0bc-4b05-4d60-968d-44b609698f40-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-96w96\" (UID: \"571cb0bc-4b05-4d60-968d-44b609698f40\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" Nov 01 22:29:10.117024 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.116953 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9wrt\" (UniqueName: \"kubernetes.io/projected/ddbdcba2-7288-42eb-a7ba-c6b57a338e47-kube-api-access-g9wrt\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pkdpb\" (UID: \"ddbdcba2-7288-42eb-a7ba-c6b57a338e47\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pkdpb" Nov 01 22:29:10.117024 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.116985 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/571cb0bc-4b05-4d60-968d-44b609698f40-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-96w96\" (UID: \"571cb0bc-4b05-4d60-968d-44b609698f40\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" Nov 01 22:29:10.117024 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.117017 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqp4s\" (UniqueName: \"kubernetes.io/projected/571cb0bc-4b05-4d60-968d-44b609698f40-kube-api-access-jqp4s\") pod \"istiod-openshift-gateway-7876bfcb74-96w96\" (UID: \"571cb0bc-4b05-4d60-968d-44b609698f40\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" Nov 01 22:29:10.117289 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.117058 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddbdcba2-7288-42eb-a7ba-c6b57a338e47-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pkdpb\" (UID: \"ddbdcba2-7288-42eb-a7ba-c6b57a338e47\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pkdpb" Nov 01 22:29:10.117289 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.117091 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/571cb0bc-4b05-4d60-968d-44b609698f40-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-96w96\" (UID: \"571cb0bc-4b05-4d60-968d-44b609698f40\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" Nov 01 22:29:10.117289 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.117148 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/571cb0bc-4b05-4d60-968d-44b609698f40-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-96w96\" (UID: \"571cb0bc-4b05-4d60-968d-44b609698f40\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" Nov 01 22:29:10.117289 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.117179 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/571cb0bc-4b05-4d60-968d-44b609698f40-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-96w96\" (UID: \"571cb0bc-4b05-4d60-968d-44b609698f40\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" Nov 01 22:29:10.118255 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.117597 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddbdcba2-7288-42eb-a7ba-c6b57a338e47-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pkdpb\" (UID: \"ddbdcba2-7288-42eb-a7ba-c6b57a338e47\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pkdpb" Nov 01 22:29:10.118255 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.118184 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddbdcba2-7288-42eb-a7ba-c6b57a338e47-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pkdpb\" (UID: \"ddbdcba2-7288-42eb-a7ba-c6b57a338e47\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pkdpb" Nov 01 22:29:10.127307 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.127265 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9wrt\" (UniqueName: \"kubernetes.io/projected/ddbdcba2-7288-42eb-a7ba-c6b57a338e47-kube-api-access-g9wrt\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pkdpb\" (UID: \"ddbdcba2-7288-42eb-a7ba-c6b57a338e47\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pkdpb" Nov 01 22:29:10.217923 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.217886 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqp4s\" (UniqueName: \"kubernetes.io/projected/571cb0bc-4b05-4d60-968d-44b609698f40-kube-api-access-jqp4s\") pod \"istiod-openshift-gateway-7876bfcb74-96w96\" (UID: \"571cb0bc-4b05-4d60-968d-44b609698f40\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" Nov 01 22:29:10.218115 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.217933 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/571cb0bc-4b05-4d60-968d-44b609698f40-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-96w96\" (UID: \"571cb0bc-4b05-4d60-968d-44b609698f40\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" Nov 01 22:29:10.218115 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.217970 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/571cb0bc-4b05-4d60-968d-44b609698f40-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-96w96\" (UID: \"571cb0bc-4b05-4d60-968d-44b609698f40\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" Nov 01 22:29:10.218115 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.217992 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/571cb0bc-4b05-4d60-968d-44b609698f40-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-96w96\" (UID: \"571cb0bc-4b05-4d60-968d-44b609698f40\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" Nov 01 22:29:10.218115 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.218022 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/571cb0bc-4b05-4d60-968d-44b609698f40-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-96w96\" (UID: \"571cb0bc-4b05-4d60-968d-44b609698f40\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" Nov 01 22:29:10.218115 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.218045 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/571cb0bc-4b05-4d60-968d-44b609698f40-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-96w96\" (UID: \"571cb0bc-4b05-4d60-968d-44b609698f40\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" Nov 01 22:29:10.218115 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.218081 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/571cb0bc-4b05-4d60-968d-44b609698f40-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-96w96\" (UID: \"571cb0bc-4b05-4d60-968d-44b609698f40\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" Nov 01 22:29:10.218862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.218795 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/571cb0bc-4b05-4d60-968d-44b609698f40-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-96w96\" (UID: \"571cb0bc-4b05-4d60-968d-44b609698f40\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" Nov 01 22:29:10.220670 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.220646 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/571cb0bc-4b05-4d60-968d-44b609698f40-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-96w96\" (UID: \"571cb0bc-4b05-4d60-968d-44b609698f40\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" Nov 01 22:29:10.220670 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.220656 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/571cb0bc-4b05-4d60-968d-44b609698f40-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-96w96\" (UID: \"571cb0bc-4b05-4d60-968d-44b609698f40\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" Nov 01 22:29:10.221044 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.221030 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/571cb0bc-4b05-4d60-968d-44b609698f40-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-96w96\" (UID: \"571cb0bc-4b05-4d60-968d-44b609698f40\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" Nov 01 22:29:10.221090 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.221030 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/571cb0bc-4b05-4d60-968d-44b609698f40-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-96w96\" (UID: \"571cb0bc-4b05-4d60-968d-44b609698f40\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" Nov 01 22:29:10.238000 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.237964 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/571cb0bc-4b05-4d60-968d-44b609698f40-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-96w96\" (UID: \"571cb0bc-4b05-4d60-968d-44b609698f40\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" Nov 01 22:29:10.238176 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.238154 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqp4s\" (UniqueName: \"kubernetes.io/projected/571cb0bc-4b05-4d60-968d-44b609698f40-kube-api-access-jqp4s\") pod \"istiod-openshift-gateway-7876bfcb74-96w96\" (UID: \"571cb0bc-4b05-4d60-968d-44b609698f40\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" Nov 01 22:29:10.251970 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.251939 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pkdpb" Nov 01 22:29:10.379045 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.378974 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qkpgg"] Nov 01 22:29:10.382386 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:29:10.382353 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7afb798_1ba6_440d_9ba2_b2da734044b2.slice/crio-c4e124264f065d529aed35d465b0a972e1eb65783d36d10fe3aa455685699b18 WatchSource:0}: Error finding container c4e124264f065d529aed35d465b0a972e1eb65783d36d10fe3aa455685699b18: Status 404 returned error can't find the container with id c4e124264f065d529aed35d465b0a972e1eb65783d36d10fe3aa455685699b18 Nov 01 22:29:10.384452 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.384438 5011 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:29:10.392391 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.392370 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" Nov 01 22:29:10.522398 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.522367 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pkdpb"] Nov 01 22:29:10.525140 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:29:10.525113 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddbdcba2_7288_42eb_a7ba_c6b57a338e47.slice/crio-38916ca43862173ccbe62e7355a24d687498ed13dc3213a31a9da8e5647ed813 WatchSource:0}: Error finding container 38916ca43862173ccbe62e7355a24d687498ed13dc3213a31a9da8e5647ed813: Status 404 returned error can't find the container with id 38916ca43862173ccbe62e7355a24d687498ed13dc3213a31a9da8e5647ed813 Nov 01 22:29:10.665999 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.665919 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96"] Nov 01 22:29:10.669201 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:29:10.669174 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod571cb0bc_4b05_4d60_968d_44b609698f40.slice/crio-cae92c7ddc2dc9d4c83226cd2353f4553deb6c6270e76610805a0e4a0f95f452 WatchSource:0}: Error finding container cae92c7ddc2dc9d4c83226cd2353f4553deb6c6270e76610805a0e4a0f95f452: Status 404 returned error can't find the container with id cae92c7ddc2dc9d4c83226cd2353f4553deb6c6270e76610805a0e4a0f95f452 Nov 01 22:29:10.952314 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.952276 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pkdpb" event={"ID":"ddbdcba2-7288-42eb-a7ba-c6b57a338e47","Type":"ContainerStarted","Data":"38916ca43862173ccbe62e7355a24d687498ed13dc3213a31a9da8e5647ed813"} Nov 01 22:29:10.953404 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.953382 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qkpgg" event={"ID":"a7afb798-1ba6-440d-9ba2-b2da734044b2","Type":"ContainerStarted","Data":"c4e124264f065d529aed35d465b0a972e1eb65783d36d10fe3aa455685699b18"} Nov 01 22:29:10.954495 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:10.954470 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" event={"ID":"571cb0bc-4b05-4d60-968d-44b609698f40","Type":"ContainerStarted","Data":"cae92c7ddc2dc9d4c83226cd2353f4553deb6c6270e76610805a0e4a0f95f452"} Nov 01 22:29:14.766230 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:14.766190 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:29:14.766607 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:14.766265 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:29:14.975251 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:14.975205 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" event={"ID":"571cb0bc-4b05-4d60-968d-44b609698f40","Type":"ContainerStarted","Data":"56fa7bd9471ba623ae9e6b45557aeb35409b8389fbe8c7cabcf055c6580e2338"} Nov 01 22:29:14.975448 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:14.975285 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" Nov 01 22:29:14.996847 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:14.996773 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" podStartSLOduration=0.902671123 podStartE2EDuration="4.996757528s" totalImagesPullingTime="4.094086405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:14.995135222 +0000 UTC m=+323.399725854" watchObservedRunningTime="2026-11-01 22:29:14.996757528 +0000 UTC m=+323.401348161" Nov 01 22:29:15.988364 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:15.988332 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" Nov 01 22:29:16.965039 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:29:16.964997 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:29:19.000986 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:19.000946 5011 generic.go:356] "Generic (PLEG): container finished" podID="a7afb798-1ba6-440d-9ba2-b2da734044b2" containerID="2bbfb395fd0c5eb79f4635f66662fb8526fbb7a3575f0db6e3f6e8811b916f11" exitCode=0 Nov 01 22:29:19.001390 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:19.001035 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qkpgg" event={"ID":"a7afb798-1ba6-440d-9ba2-b2da734044b2","Type":"ContainerDied","Data":"2bbfb395fd0c5eb79f4635f66662fb8526fbb7a3575f0db6e3f6e8811b916f11"} Nov 01 22:29:21.011072 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:21.011037 5011 generic.go:356] "Generic (PLEG): container finished" podID="ddbdcba2-7288-42eb-a7ba-c6b57a338e47" containerID="6d2fecb969286720de4c9effba7d8802ce9bd6ca1d87dd74713a58ebc7d55d6d" exitCode=0 Nov 01 22:29:21.011424 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:21.011136 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pkdpb" event={"ID":"ddbdcba2-7288-42eb-a7ba-c6b57a338e47","Type":"ContainerDied","Data":"6d2fecb969286720de4c9effba7d8802ce9bd6ca1d87dd74713a58ebc7d55d6d"} Nov 01 22:29:22.016984 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:22.016948 5011 generic.go:356] "Generic (PLEG): container finished" podID="ddbdcba2-7288-42eb-a7ba-c6b57a338e47" containerID="af832a0c94863dd0953cf00dbae1cfe666dcfb606e9975e6eaff4bdb1320991e" exitCode=0 Nov 01 22:29:22.017421 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:22.017040 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pkdpb" event={"ID":"ddbdcba2-7288-42eb-a7ba-c6b57a338e47","Type":"ContainerDied","Data":"af832a0c94863dd0953cf00dbae1cfe666dcfb606e9975e6eaff4bdb1320991e"} Nov 01 22:29:23.023269 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:23.023228 5011 generic.go:356] "Generic (PLEG): container finished" podID="a7afb798-1ba6-440d-9ba2-b2da734044b2" containerID="42ff4953302e947ef9d98e2d7cedab03bf19da5db629402b783bc08301f4bc52" exitCode=0 Nov 01 22:29:23.023676 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:23.023282 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qkpgg" event={"ID":"a7afb798-1ba6-440d-9ba2-b2da734044b2","Type":"ContainerDied","Data":"42ff4953302e947ef9d98e2d7cedab03bf19da5db629402b783bc08301f4bc52"} Nov 01 22:29:29.052955 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:29.052912 5011 generic.go:356] "Generic (PLEG): container finished" podID="ddbdcba2-7288-42eb-a7ba-c6b57a338e47" containerID="85d662604601593035f5d36e8d777b2d66abffbd9df3f0b5882da0d58fedffd1" exitCode=0 Nov 01 22:29:29.053429 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:29.053016 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pkdpb" event={"ID":"ddbdcba2-7288-42eb-a7ba-c6b57a338e47","Type":"ContainerDied","Data":"85d662604601593035f5d36e8d777b2d66abffbd9df3f0b5882da0d58fedffd1"} Nov 01 22:29:30.188558 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:30.188534 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pkdpb" Nov 01 22:29:30.308480 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:30.308419 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddbdcba2-7288-42eb-a7ba-c6b57a338e47-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddbdcba2-7288-42eb-a7ba-c6b57a338e47-bundle\") pod \"ddbdcba2-7288-42eb-a7ba-c6b57a338e47\" (UID: \"ddbdcba2-7288-42eb-a7ba-c6b57a338e47\") " Nov 01 22:29:30.308480 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:30.308477 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddbdcba2-7288-42eb-a7ba-c6b57a338e47-kube-api-access-g9wrt\" (UniqueName: \"kubernetes.io/projected/ddbdcba2-7288-42eb-a7ba-c6b57a338e47-kube-api-access-g9wrt\") pod \"ddbdcba2-7288-42eb-a7ba-c6b57a338e47\" (UID: \"ddbdcba2-7288-42eb-a7ba-c6b57a338e47\") " Nov 01 22:29:30.308618 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:30.308548 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddbdcba2-7288-42eb-a7ba-c6b57a338e47-util\" (UniqueName: \"kubernetes.io/empty-dir/ddbdcba2-7288-42eb-a7ba-c6b57a338e47-util\") pod \"ddbdcba2-7288-42eb-a7ba-c6b57a338e47\" (UID: \"ddbdcba2-7288-42eb-a7ba-c6b57a338e47\") " Nov 01 22:29:30.308927 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:30.308903 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbdcba2-7288-42eb-a7ba-c6b57a338e47-bundle" pod "ddbdcba2-7288-42eb-a7ba-c6b57a338e47" (UID: "ddbdcba2-7288-42eb-a7ba-c6b57a338e47"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:30.310708 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:30.310692 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddbdcba2-7288-42eb-a7ba-c6b57a338e47-kube-api-access-g9wrt" pod "ddbdcba2-7288-42eb-a7ba-c6b57a338e47" (UID: "ddbdcba2-7288-42eb-a7ba-c6b57a338e47"). InnerVolumeSpecName "kube-api-access-g9wrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:29:30.312740 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:30.312718 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbdcba2-7288-42eb-a7ba-c6b57a338e47-util" pod "ddbdcba2-7288-42eb-a7ba-c6b57a338e47" (UID: "ddbdcba2-7288-42eb-a7ba-c6b57a338e47"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:30.409745 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:30.409719 5011 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddbdcba2-7288-42eb-a7ba-c6b57a338e47-bundle\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:30.409745 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:30.409747 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9wrt\" (UniqueName: \"kubernetes.io/projected/ddbdcba2-7288-42eb-a7ba-c6b57a338e47-kube-api-access-g9wrt\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:30.409914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:30.409757 5011 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddbdcba2-7288-42eb-a7ba-c6b57a338e47-util\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:31.062099 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:31.062067 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pkdpb" Nov 01 22:29:31.062099 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:31.062085 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pkdpb" event={"ID":"ddbdcba2-7288-42eb-a7ba-c6b57a338e47","Type":"ContainerDied","Data":"38916ca43862173ccbe62e7355a24d687498ed13dc3213a31a9da8e5647ed813"} Nov 01 22:29:31.062307 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:31.062129 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38916ca43862173ccbe62e7355a24d687498ed13dc3213a31a9da8e5647ed813" Nov 01 22:29:31.070111 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:31.070077 5011 generic.go:356] "Generic (PLEG): container finished" podID="a7afb798-1ba6-440d-9ba2-b2da734044b2" containerID="45548dd97dd762645a8c119cc45f7abee080a920d63db14cb13a2d5a671dbf04" exitCode=0 Nov 01 22:29:31.070271 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:31.070118 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qkpgg" event={"ID":"a7afb798-1ba6-440d-9ba2-b2da734044b2","Type":"ContainerDied","Data":"45548dd97dd762645a8c119cc45f7abee080a920d63db14cb13a2d5a671dbf04"} Nov 01 22:29:32.214205 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:32.214169 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qkpgg" Nov 01 22:29:32.325331 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:32.325246 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7afb798-1ba6-440d-9ba2-b2da734044b2-util\" (UniqueName: \"kubernetes.io/empty-dir/a7afb798-1ba6-440d-9ba2-b2da734044b2-util\") pod \"a7afb798-1ba6-440d-9ba2-b2da734044b2\" (UID: \"a7afb798-1ba6-440d-9ba2-b2da734044b2\") " Nov 01 22:29:32.325331 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:32.325301 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7afb798-1ba6-440d-9ba2-b2da734044b2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7afb798-1ba6-440d-9ba2-b2da734044b2-bundle\") pod \"a7afb798-1ba6-440d-9ba2-b2da734044b2\" (UID: \"a7afb798-1ba6-440d-9ba2-b2da734044b2\") " Nov 01 22:29:32.325533 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:32.325340 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7afb798-1ba6-440d-9ba2-b2da734044b2-kube-api-access-7jfbd\" (UniqueName: \"kubernetes.io/projected/a7afb798-1ba6-440d-9ba2-b2da734044b2-kube-api-access-7jfbd\") pod \"a7afb798-1ba6-440d-9ba2-b2da734044b2\" (UID: \"a7afb798-1ba6-440d-9ba2-b2da734044b2\") " Nov 01 22:29:32.325777 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:32.325750 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7afb798-1ba6-440d-9ba2-b2da734044b2-bundle" pod "a7afb798-1ba6-440d-9ba2-b2da734044b2" (UID: "a7afb798-1ba6-440d-9ba2-b2da734044b2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:32.327624 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:32.327599 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7afb798-1ba6-440d-9ba2-b2da734044b2-kube-api-access-7jfbd" pod "a7afb798-1ba6-440d-9ba2-b2da734044b2" (UID: "a7afb798-1ba6-440d-9ba2-b2da734044b2"). InnerVolumeSpecName "kube-api-access-7jfbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:29:32.329241 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:32.329216 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7afb798-1ba6-440d-9ba2-b2da734044b2-util" pod "a7afb798-1ba6-440d-9ba2-b2da734044b2" (UID: "a7afb798-1ba6-440d-9ba2-b2da734044b2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:29:32.425973 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:32.425936 5011 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7afb798-1ba6-440d-9ba2-b2da734044b2-util\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:32.425973 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:32.425967 5011 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7afb798-1ba6-440d-9ba2-b2da734044b2-bundle\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:32.425973 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:32.425978 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jfbd\" (UniqueName: \"kubernetes.io/projected/a7afb798-1ba6-440d-9ba2-b2da734044b2-kube-api-access-7jfbd\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:29:33.079919 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:33.079881 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qkpgg" event={"ID":"a7afb798-1ba6-440d-9ba2-b2da734044b2","Type":"ContainerDied","Data":"c4e124264f065d529aed35d465b0a972e1eb65783d36d10fe3aa455685699b18"} Nov 01 22:29:33.080088 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:33.079942 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4e124264f065d529aed35d465b0a972e1eb65783d36d10fe3aa455685699b18" Nov 01 22:29:33.080088 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:29:33.079896 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qkpgg" Nov 01 22:29:46.958893 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:29:46.958854 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:30:09.071675 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.071594 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-qqvnv"] Nov 01 22:30:09.072198 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.072012 5011 cpu_manager.go:414] "containerMap: removing container" podUID="a7afb798-1ba6-440d-9ba2-b2da734044b2" containerName="util" Nov 01 22:30:09.072198 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.072030 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7afb798-1ba6-440d-9ba2-b2da734044b2" containerName="util" Nov 01 22:30:09.072198 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.072048 5011 cpu_manager.go:414] "containerMap: removing container" podUID="a7afb798-1ba6-440d-9ba2-b2da734044b2" containerName="extract" Nov 01 22:30:09.072198 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.072055 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7afb798-1ba6-440d-9ba2-b2da734044b2" containerName="extract" Nov 01 22:30:09.072198 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.072070 5011 cpu_manager.go:414] "containerMap: removing container" podUID="ddbdcba2-7288-42eb-a7ba-c6b57a338e47" containerName="util" Nov 01 22:30:09.072198 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.072077 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddbdcba2-7288-42eb-a7ba-c6b57a338e47" containerName="util" Nov 01 22:30:09.072198 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.072100 5011 cpu_manager.go:414] "containerMap: removing container" podUID="a7afb798-1ba6-440d-9ba2-b2da734044b2" containerName="pull" Nov 01 22:30:09.072198 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.072106 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7afb798-1ba6-440d-9ba2-b2da734044b2" containerName="pull" Nov 01 22:30:09.072198 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.072120 5011 cpu_manager.go:414] "containerMap: removing container" podUID="ddbdcba2-7288-42eb-a7ba-c6b57a338e47" containerName="pull" Nov 01 22:30:09.072198 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.072127 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddbdcba2-7288-42eb-a7ba-c6b57a338e47" containerName="pull" Nov 01 22:30:09.072198 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.072136 5011 cpu_manager.go:414] "containerMap: removing container" podUID="ddbdcba2-7288-42eb-a7ba-c6b57a338e47" containerName="extract" Nov 01 22:30:09.072198 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.072143 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddbdcba2-7288-42eb-a7ba-c6b57a338e47" containerName="extract" Nov 01 22:30:09.072826 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.072218 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddbdcba2-7288-42eb-a7ba-c6b57a338e47" containerName="extract" Nov 01 22:30:09.072826 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.072232 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7afb798-1ba6-440d-9ba2-b2da734044b2" containerName="extract" Nov 01 22:30:09.074284 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.074266 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-qqvnv" Nov 01 22:30:09.111227 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.111196 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-qqvnv"] Nov 01 22:30:09.115243 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.115219 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44fa05ae-24ca-4cc3-954d-47307ffdbfae-tmp\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-qqvnv\" (UID: \"44fa05ae-24ca-4cc3-954d-47307ffdbfae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-qqvnv" Nov 01 22:30:09.115392 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.115256 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/44fa05ae-24ca-4cc3-954d-47307ffdbfae-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-qqvnv\" (UID: \"44fa05ae-24ca-4cc3-954d-47307ffdbfae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-qqvnv" Nov 01 22:30:09.115392 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.115284 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44fa05ae-24ca-4cc3-954d-47307ffdbfae-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-qqvnv\" (UID: \"44fa05ae-24ca-4cc3-954d-47307ffdbfae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-qqvnv" Nov 01 22:30:09.115488 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.115400 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74nr4\" (UniqueName: \"kubernetes.io/projected/44fa05ae-24ca-4cc3-954d-47307ffdbfae-kube-api-access-74nr4\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-qqvnv\" (UID: \"44fa05ae-24ca-4cc3-954d-47307ffdbfae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-qqvnv" Nov 01 22:30:09.115488 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.115449 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/44fa05ae-24ca-4cc3-954d-47307ffdbfae-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-qqvnv\" (UID: \"44fa05ae-24ca-4cc3-954d-47307ffdbfae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-qqvnv" Nov 01 22:30:09.115488 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.115472 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/44fa05ae-24ca-4cc3-954d-47307ffdbfae-manifests\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-qqvnv\" (UID: \"44fa05ae-24ca-4cc3-954d-47307ffdbfae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-qqvnv" Nov 01 22:30:09.160191 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.160157 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 01 22:30:09.160191 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.160158 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 01 22:30:09.160420 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.160294 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 01 22:30:09.160420 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.160332 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 01 22:30:09.160420 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.160359 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-bqmh4\"" Nov 01 22:30:09.160420 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.160420 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 01 22:30:09.216192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.216163 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74nr4\" (UniqueName: \"kubernetes.io/projected/44fa05ae-24ca-4cc3-954d-47307ffdbfae-kube-api-access-74nr4\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-qqvnv\" (UID: \"44fa05ae-24ca-4cc3-954d-47307ffdbfae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-qqvnv" Nov 01 22:30:09.216394 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.216201 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/44fa05ae-24ca-4cc3-954d-47307ffdbfae-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-qqvnv\" (UID: \"44fa05ae-24ca-4cc3-954d-47307ffdbfae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-qqvnv" Nov 01 22:30:09.216394 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.216229 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/44fa05ae-24ca-4cc3-954d-47307ffdbfae-manifests\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-qqvnv\" (UID: \"44fa05ae-24ca-4cc3-954d-47307ffdbfae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-qqvnv" Nov 01 22:30:09.216394 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.216287 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44fa05ae-24ca-4cc3-954d-47307ffdbfae-tmp\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-qqvnv\" (UID: \"44fa05ae-24ca-4cc3-954d-47307ffdbfae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-qqvnv" Nov 01 22:30:09.216394 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.216323 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/44fa05ae-24ca-4cc3-954d-47307ffdbfae-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-qqvnv\" (UID: \"44fa05ae-24ca-4cc3-954d-47307ffdbfae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-qqvnv" Nov 01 22:30:09.216394 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.216351 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44fa05ae-24ca-4cc3-954d-47307ffdbfae-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-qqvnv\" (UID: \"44fa05ae-24ca-4cc3-954d-47307ffdbfae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-qqvnv" Nov 01 22:30:09.216849 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.216796 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/44fa05ae-24ca-4cc3-954d-47307ffdbfae-manifests\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-qqvnv\" (UID: \"44fa05ae-24ca-4cc3-954d-47307ffdbfae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-qqvnv" Nov 01 22:30:09.219022 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.218993 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/44fa05ae-24ca-4cc3-954d-47307ffdbfae-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-qqvnv\" (UID: \"44fa05ae-24ca-4cc3-954d-47307ffdbfae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-qqvnv" Nov 01 22:30:09.219022 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.219015 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/44fa05ae-24ca-4cc3-954d-47307ffdbfae-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-qqvnv\" (UID: \"44fa05ae-24ca-4cc3-954d-47307ffdbfae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-qqvnv" Nov 01 22:30:09.219194 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.219156 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44fa05ae-24ca-4cc3-954d-47307ffdbfae-tmp\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-qqvnv\" (UID: \"44fa05ae-24ca-4cc3-954d-47307ffdbfae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-qqvnv" Nov 01 22:30:09.219272 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.219207 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44fa05ae-24ca-4cc3-954d-47307ffdbfae-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-qqvnv\" (UID: \"44fa05ae-24ca-4cc3-954d-47307ffdbfae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-qqvnv" Nov 01 22:30:09.227531 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.227509 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74nr4\" (UniqueName: \"kubernetes.io/projected/44fa05ae-24ca-4cc3-954d-47307ffdbfae-kube-api-access-74nr4\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-qqvnv\" (UID: \"44fa05ae-24ca-4cc3-954d-47307ffdbfae\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-qqvnv" Nov 01 22:30:09.387002 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.386915 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-qqvnv" Nov 01 22:30:09.683396 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:09.683351 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-qqvnv"] Nov 01 22:30:09.685705 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:30:09.685671 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44fa05ae_24ca_4cc3_954d_47307ffdbfae.slice/crio-9be3e6c8a2e3eac5aa38688c729ca1cba8545b0bdffafdf5981c7917f316300b WatchSource:0}: Error finding container 9be3e6c8a2e3eac5aa38688c729ca1cba8545b0bdffafdf5981c7917f316300b: Status 404 returned error can't find the container with id 9be3e6c8a2e3eac5aa38688c729ca1cba8545b0bdffafdf5981c7917f316300b Nov 01 22:30:10.211723 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:10.211687 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-qqvnv" event={"ID":"44fa05ae-24ca-4cc3-954d-47307ffdbfae","Type":"ContainerStarted","Data":"9be3e6c8a2e3eac5aa38688c729ca1cba8545b0bdffafdf5981c7917f316300b"} Nov 01 22:30:14.229853 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:14.229742 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-qqvnv" event={"ID":"44fa05ae-24ca-4cc3-954d-47307ffdbfae","Type":"ContainerStarted","Data":"7c21964e335ef119c72542da602e7d7121fdead636f1b735c436209fd4322ad4"} Nov 01 22:30:15.234919 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:15.234884 5011 generic.go:356] "Generic (PLEG): container finished" podID="44fa05ae-24ca-4cc3-954d-47307ffdbfae" containerID="7c21964e335ef119c72542da602e7d7121fdead636f1b735c436209fd4322ad4" exitCode=0 Nov 01 22:30:15.235317 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:15.234972 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-qqvnv" event={"ID":"44fa05ae-24ca-4cc3-954d-47307ffdbfae","Type":"ContainerDied","Data":"7c21964e335ef119c72542da602e7d7121fdead636f1b735c436209fd4322ad4"} Nov 01 22:30:16.241125 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:16.241088 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-qqvnv" event={"ID":"44fa05ae-24ca-4cc3-954d-47307ffdbfae","Type":"ContainerStarted","Data":"bbe1e3753a6f13230dd7c5b229f3b593ac734af68d54fed4a13f8272f056470a"} Nov 01 22:30:16.241526 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:16.241201 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-qqvnv" Nov 01 22:30:16.958569 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:30:16.958526 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:30:27.247735 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:27.247704 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-qqvnv" Nov 01 22:30:27.284638 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:27.284592 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-qqvnv" podStartSLOduration=13.187393532 podStartE2EDuration="18.284577083s" totalImagesPullingTime="4.650160128s" totalInitContainerRuntime="447.023423ms" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:16.403858386 +0000 UTC m=+384.808449014" watchObservedRunningTime="2026-11-01 22:30:27.284577083 +0000 UTC m=+395.689167715" Nov 01 22:30:40.399232 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:40.399198 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-9bc8df847-chtf8"] Nov 01 22:30:40.402447 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:40.402423 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9bc8df847-chtf8" Nov 01 22:30:40.421503 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:40.421473 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9bc8df847-chtf8"] Nov 01 22:30:40.470414 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:40.470379 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4kkj\" (UniqueName: \"kubernetes.io/projected/199b342c-2f8e-43eb-a1dd-da546aa1059e-kube-api-access-t4kkj\") pod \"kube-auth-proxy-9bc8df847-chtf8\" (UID: \"199b342c-2f8e-43eb-a1dd-da546aa1059e\") " pod="openshift-ingress/kube-auth-proxy-9bc8df847-chtf8" Nov 01 22:30:40.470582 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:40.470426 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/199b342c-2f8e-43eb-a1dd-da546aa1059e-tls-certs\") pod \"kube-auth-proxy-9bc8df847-chtf8\" (UID: \"199b342c-2f8e-43eb-a1dd-da546aa1059e\") " pod="openshift-ingress/kube-auth-proxy-9bc8df847-chtf8" Nov 01 22:30:40.470582 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:40.470531 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/199b342c-2f8e-43eb-a1dd-da546aa1059e-tmp\") pod \"kube-auth-proxy-9bc8df847-chtf8\" (UID: \"199b342c-2f8e-43eb-a1dd-da546aa1059e\") " pod="openshift-ingress/kube-auth-proxy-9bc8df847-chtf8" Nov 01 22:30:40.485615 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:40.485576 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 01 22:30:40.485940 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:40.485919 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 01 22:30:40.486099 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:40.485931 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-875b7\"" Nov 01 22:30:40.571758 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:40.571721 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4kkj\" (UniqueName: \"kubernetes.io/projected/199b342c-2f8e-43eb-a1dd-da546aa1059e-kube-api-access-t4kkj\") pod \"kube-auth-proxy-9bc8df847-chtf8\" (UID: \"199b342c-2f8e-43eb-a1dd-da546aa1059e\") " pod="openshift-ingress/kube-auth-proxy-9bc8df847-chtf8" Nov 01 22:30:40.572039 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:40.572019 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/199b342c-2f8e-43eb-a1dd-da546aa1059e-tls-certs\") pod \"kube-auth-proxy-9bc8df847-chtf8\" (UID: \"199b342c-2f8e-43eb-a1dd-da546aa1059e\") " pod="openshift-ingress/kube-auth-proxy-9bc8df847-chtf8" Nov 01 22:30:40.572202 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:40.572178 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/199b342c-2f8e-43eb-a1dd-da546aa1059e-tmp\") pod \"kube-auth-proxy-9bc8df847-chtf8\" (UID: \"199b342c-2f8e-43eb-a1dd-da546aa1059e\") " pod="openshift-ingress/kube-auth-proxy-9bc8df847-chtf8" Nov 01 22:30:40.575625 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:40.575590 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/199b342c-2f8e-43eb-a1dd-da546aa1059e-tls-certs\") pod \"kube-auth-proxy-9bc8df847-chtf8\" (UID: \"199b342c-2f8e-43eb-a1dd-da546aa1059e\") " pod="openshift-ingress/kube-auth-proxy-9bc8df847-chtf8" Nov 01 22:30:40.578096 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:40.578070 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/199b342c-2f8e-43eb-a1dd-da546aa1059e-tmp\") pod \"kube-auth-proxy-9bc8df847-chtf8\" (UID: \"199b342c-2f8e-43eb-a1dd-da546aa1059e\") " pod="openshift-ingress/kube-auth-proxy-9bc8df847-chtf8" Nov 01 22:30:40.643273 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:40.643239 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4kkj\" (UniqueName: \"kubernetes.io/projected/199b342c-2f8e-43eb-a1dd-da546aa1059e-kube-api-access-t4kkj\") pod \"kube-auth-proxy-9bc8df847-chtf8\" (UID: \"199b342c-2f8e-43eb-a1dd-da546aa1059e\") " pod="openshift-ingress/kube-auth-proxy-9bc8df847-chtf8" Nov 01 22:30:40.715117 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:40.715083 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9bc8df847-chtf8" Nov 01 22:30:41.057351 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:41.057271 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9bc8df847-chtf8"] Nov 01 22:30:41.063191 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:30:41.063160 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod199b342c_2f8e_43eb_a1dd_da546aa1059e.slice/crio-2f86ff805e7922c587be42da7847234e9ac21bf04642b824aff3d72c88f1d335 WatchSource:0}: Error finding container 2f86ff805e7922c587be42da7847234e9ac21bf04642b824aff3d72c88f1d335: Status 404 returned error can't find the container with id 2f86ff805e7922c587be42da7847234e9ac21bf04642b824aff3d72c88f1d335 Nov 01 22:30:41.339732 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:41.339645 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9bc8df847-chtf8" event={"ID":"199b342c-2f8e-43eb-a1dd-da546aa1059e","Type":"ContainerStarted","Data":"2f86ff805e7922c587be42da7847234e9ac21bf04642b824aff3d72c88f1d335"} Nov 01 22:30:42.330823 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:42.330789 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:30:42.334445 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:42.334429 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-54nss" Nov 01 22:30:42.349378 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:42.349355 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 01 22:30:42.349378 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:42.349368 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 01 22:30:42.349591 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:42.349404 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 01 22:30:42.350509 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:42.350484 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:30:42.351400 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:42.351383 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2m2x7\"" Nov 01 22:30:42.433962 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:42.433923 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:30:42.490004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:42.489970 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5da97fc4-69c1-4cba-b7d6-9a85b49fb23f-config-file\") pod \"limitador-limitador-7c9b87d574-54nss\" (UID: \"5da97fc4-69c1-4cba-b7d6-9a85b49fb23f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-54nss" Nov 01 22:30:42.490180 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:42.490011 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mg7j\" (UniqueName: \"kubernetes.io/projected/5da97fc4-69c1-4cba-b7d6-9a85b49fb23f-kube-api-access-6mg7j\") pod \"limitador-limitador-7c9b87d574-54nss\" (UID: \"5da97fc4-69c1-4cba-b7d6-9a85b49fb23f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-54nss" Nov 01 22:30:42.590513 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:42.590426 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5da97fc4-69c1-4cba-b7d6-9a85b49fb23f-config-file\") pod \"limitador-limitador-7c9b87d574-54nss\" (UID: \"5da97fc4-69c1-4cba-b7d6-9a85b49fb23f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-54nss" Nov 01 22:30:42.590513 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:42.590466 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mg7j\" (UniqueName: \"kubernetes.io/projected/5da97fc4-69c1-4cba-b7d6-9a85b49fb23f-kube-api-access-6mg7j\") pod \"limitador-limitador-7c9b87d574-54nss\" (UID: \"5da97fc4-69c1-4cba-b7d6-9a85b49fb23f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-54nss" Nov 01 22:30:42.591176 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:42.591151 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5da97fc4-69c1-4cba-b7d6-9a85b49fb23f-config-file\") pod \"limitador-limitador-7c9b87d574-54nss\" (UID: \"5da97fc4-69c1-4cba-b7d6-9a85b49fb23f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-54nss" Nov 01 22:30:42.607650 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:42.607589 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mg7j\" (UniqueName: \"kubernetes.io/projected/5da97fc4-69c1-4cba-b7d6-9a85b49fb23f-kube-api-access-6mg7j\") pod \"limitador-limitador-7c9b87d574-54nss\" (UID: \"5da97fc4-69c1-4cba-b7d6-9a85b49fb23f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-54nss" Nov 01 22:30:42.650284 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:42.650252 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-54nss" Nov 01 22:30:42.953341 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:42.953314 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:30:42.955613 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:30:42.955581 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5da97fc4_69c1_4cba_b7d6_9a85b49fb23f.slice/crio-b3f50d02e41637a3baba4d8de440a40fd67863a35ae60a554df31c252e88a000 WatchSource:0}: Error finding container b3f50d02e41637a3baba4d8de440a40fd67863a35ae60a554df31c252e88a000: Status 404 returned error can't find the container with id b3f50d02e41637a3baba4d8de440a40fd67863a35ae60a554df31c252e88a000 Nov 01 22:30:43.047591 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:43.047558 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rwmtq"] Nov 01 22:30:43.052777 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:43.051986 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-rwmtq" Nov 01 22:30:43.058290 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:43.058262 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qzl2p\"" Nov 01 22:30:43.063215 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:43.063191 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rwmtq"] Nov 01 22:30:43.181291 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:43.181254 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9d2z2"] Nov 01 22:30:43.186137 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:43.186059 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-9d2z2" Nov 01 22:30:43.194937 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:43.194911 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9d2z2"] Nov 01 22:30:43.195395 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:43.195373 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls59b\" (UniqueName: \"kubernetes.io/projected/ddb4396c-e40c-4257-b10a-87f228f3d7e9-kube-api-access-ls59b\") pod \"authorino-57f7f8659f-rwmtq\" (UID: \"ddb4396c-e40c-4257-b10a-87f228f3d7e9\") " pod="kuadrant-system/authorino-57f7f8659f-rwmtq" Nov 01 22:30:43.296015 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:43.295925 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9hgf\" (UniqueName: \"kubernetes.io/projected/956c392a-258c-4c1c-aada-74d72921ecc2-kube-api-access-n9hgf\") pod \"authorino-56c6c9ddd5-9d2z2\" (UID: \"956c392a-258c-4c1c-aada-74d72921ecc2\") " pod="kuadrant-system/authorino-56c6c9ddd5-9d2z2" Nov 01 22:30:43.296188 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:43.296065 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls59b\" (UniqueName: \"kubernetes.io/projected/ddb4396c-e40c-4257-b10a-87f228f3d7e9-kube-api-access-ls59b\") pod \"authorino-57f7f8659f-rwmtq\" (UID: \"ddb4396c-e40c-4257-b10a-87f228f3d7e9\") " pod="kuadrant-system/authorino-57f7f8659f-rwmtq" Nov 01 22:30:43.305039 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:43.305010 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls59b\" (UniqueName: \"kubernetes.io/projected/ddb4396c-e40c-4257-b10a-87f228f3d7e9-kube-api-access-ls59b\") pod \"authorino-57f7f8659f-rwmtq\" (UID: \"ddb4396c-e40c-4257-b10a-87f228f3d7e9\") " pod="kuadrant-system/authorino-57f7f8659f-rwmtq" Nov 01 22:30:43.349892 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:43.349859 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-54nss" event={"ID":"5da97fc4-69c1-4cba-b7d6-9a85b49fb23f","Type":"ContainerStarted","Data":"b3f50d02e41637a3baba4d8de440a40fd67863a35ae60a554df31c252e88a000"} Nov 01 22:30:43.373665 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:43.373634 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-rwmtq" Nov 01 22:30:43.396617 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:43.396589 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9hgf\" (UniqueName: \"kubernetes.io/projected/956c392a-258c-4c1c-aada-74d72921ecc2-kube-api-access-n9hgf\") pod \"authorino-56c6c9ddd5-9d2z2\" (UID: \"956c392a-258c-4c1c-aada-74d72921ecc2\") " pod="kuadrant-system/authorino-56c6c9ddd5-9d2z2" Nov 01 22:30:43.421474 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:43.421217 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9hgf\" (UniqueName: \"kubernetes.io/projected/956c392a-258c-4c1c-aada-74d72921ecc2-kube-api-access-n9hgf\") pod \"authorino-56c6c9ddd5-9d2z2\" (UID: \"956c392a-258c-4c1c-aada-74d72921ecc2\") " pod="kuadrant-system/authorino-56c6c9ddd5-9d2z2" Nov 01 22:30:43.499713 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:43.499677 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-9d2z2" Nov 01 22:30:43.650918 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:43.650678 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rwmtq"] Nov 01 22:30:43.655983 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:30:43.655950 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podddb4396c_e40c_4257_b10a_87f228f3d7e9.slice/crio-4fd33edd8dac6368541d3e9c30b0352bfcd5605ceccd17f48d4458fab09b15bb WatchSource:0}: Error finding container 4fd33edd8dac6368541d3e9c30b0352bfcd5605ceccd17f48d4458fab09b15bb: Status 404 returned error can't find the container with id 4fd33edd8dac6368541d3e9c30b0352bfcd5605ceccd17f48d4458fab09b15bb Nov 01 22:30:43.810728 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:43.810705 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9d2z2"] Nov 01 22:30:43.813676 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:30:43.813645 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod956c392a_258c_4c1c_aada_74d72921ecc2.slice/crio-dbda4b69480002b4e1dffe5f09b8a72e6a043d93bf33ba7db3f928e219fec28c WatchSource:0}: Error finding container dbda4b69480002b4e1dffe5f09b8a72e6a043d93bf33ba7db3f928e219fec28c: Status 404 returned error can't find the container with id dbda4b69480002b4e1dffe5f09b8a72e6a043d93bf33ba7db3f928e219fec28c Nov 01 22:30:44.356517 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:44.356482 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-rwmtq" event={"ID":"ddb4396c-e40c-4257-b10a-87f228f3d7e9","Type":"ContainerStarted","Data":"4fd33edd8dac6368541d3e9c30b0352bfcd5605ceccd17f48d4458fab09b15bb"} Nov 01 22:30:44.358057 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:44.358025 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-9d2z2" event={"ID":"956c392a-258c-4c1c-aada-74d72921ecc2","Type":"ContainerStarted","Data":"dbda4b69480002b4e1dffe5f09b8a72e6a043d93bf33ba7db3f928e219fec28c"} Nov 01 22:30:46.373755 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:46.373715 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9bc8df847-chtf8" event={"ID":"199b342c-2f8e-43eb-a1dd-da546aa1059e","Type":"ContainerStarted","Data":"279435d973ff3a4ead5b839ce561f4d5261edb3b4c4b805b3d9cadf97177baef"} Nov 01 22:30:46.977654 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:30:46.977617 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:30:48.384863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:48.384805 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-54nss" event={"ID":"5da97fc4-69c1-4cba-b7d6-9a85b49fb23f","Type":"ContainerStarted","Data":"8c2d0230b522186ddcde290d29c6894771b78f0a712fe5b6e49eb156fae787a4"} Nov 01 22:30:48.385305 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:48.384933 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-54nss" Nov 01 22:30:48.386435 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:48.386415 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-9d2z2" event={"ID":"956c392a-258c-4c1c-aada-74d72921ecc2","Type":"ContainerStarted","Data":"559b97c81510dc9d8d8647cbd2ccaefc31b17a70fd091076329fdcae97ce7934"} Nov 01 22:30:48.387905 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:48.387885 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-rwmtq" event={"ID":"ddb4396c-e40c-4257-b10a-87f228f3d7e9","Type":"ContainerStarted","Data":"170d46e91f9bd9e30ff744c342939c658c3044ba07f6d7c3f8d52892faae84b8"} Nov 01 22:30:48.428772 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:48.428713 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-54nss" podStartSLOduration=1.38443431 podStartE2EDuration="6.428693945s" totalImagesPullingTime="5.044259635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:48.427098382 +0000 UTC m=+416.831689034" watchObservedRunningTime="2026-11-01 22:30:48.428693945 +0000 UTC m=+416.833284578" Nov 01 22:30:48.429183 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:48.429144 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-9bc8df847-chtf8" podStartSLOduration=3.916656941 podStartE2EDuration="8.429128876s" totalImagesPullingTime="4.512471935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:46.408048522 +0000 UTC m=+414.812639186" watchObservedRunningTime="2026-11-01 22:30:48.429128876 +0000 UTC m=+416.833719508" Nov 01 22:30:48.445968 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:48.445922 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-9d2z2" podStartSLOduration=1.40011014 podStartE2EDuration="5.44590772s" totalImagesPullingTime="4.04579758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:48.445223556 +0000 UTC m=+416.849814187" watchObservedRunningTime="2026-11-01 22:30:48.44590772 +0000 UTC m=+416.850498401" Nov 01 22:30:48.473888 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:48.472136 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-rwmtq" podStartSLOduration=1.4679506070000001 podStartE2EDuration="5.472116353s" totalImagesPullingTime="4.004165746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:48.470189165 +0000 UTC m=+416.874779798" watchObservedRunningTime="2026-11-01 22:30:48.472116353 +0000 UTC m=+416.876706985" Nov 01 22:30:48.531432 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:48.531399 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rwmtq"] Nov 01 22:30:50.396161 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:50.396109 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-rwmtq" podUID="ddb4396c-e40c-4257-b10a-87f228f3d7e9" containerName="authorino" containerID="cri-o://170d46e91f9bd9e30ff744c342939c658c3044ba07f6d7c3f8d52892faae84b8" gracePeriod=30 Nov 01 22:30:50.640535 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:50.640504 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-rwmtq" Nov 01 22:30:50.769121 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:50.769087 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddb4396c-e40c-4257-b10a-87f228f3d7e9-kube-api-access-ls59b\" (UniqueName: \"kubernetes.io/projected/ddb4396c-e40c-4257-b10a-87f228f3d7e9-kube-api-access-ls59b\") pod \"ddb4396c-e40c-4257-b10a-87f228f3d7e9\" (UID: \"ddb4396c-e40c-4257-b10a-87f228f3d7e9\") " Nov 01 22:30:50.771461 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:50.771427 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddb4396c-e40c-4257-b10a-87f228f3d7e9-kube-api-access-ls59b" pod "ddb4396c-e40c-4257-b10a-87f228f3d7e9" (UID: "ddb4396c-e40c-4257-b10a-87f228f3d7e9"). InnerVolumeSpecName "kube-api-access-ls59b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:30:50.870198 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:50.870157 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls59b\" (UniqueName: \"kubernetes.io/projected/ddb4396c-e40c-4257-b10a-87f228f3d7e9-kube-api-access-ls59b\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:30:51.401374 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:51.401343 5011 generic.go:356] "Generic (PLEG): container finished" podID="ddb4396c-e40c-4257-b10a-87f228f3d7e9" containerID="170d46e91f9bd9e30ff744c342939c658c3044ba07f6d7c3f8d52892faae84b8" exitCode=0 Nov 01 22:30:51.401752 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:51.401390 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-rwmtq" Nov 01 22:30:51.401752 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:51.401430 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-rwmtq" event={"ID":"ddb4396c-e40c-4257-b10a-87f228f3d7e9","Type":"ContainerDied","Data":"170d46e91f9bd9e30ff744c342939c658c3044ba07f6d7c3f8d52892faae84b8"} Nov 01 22:30:51.401752 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:51.401484 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-rwmtq" event={"ID":"ddb4396c-e40c-4257-b10a-87f228f3d7e9","Type":"ContainerDied","Data":"4fd33edd8dac6368541d3e9c30b0352bfcd5605ceccd17f48d4458fab09b15bb"} Nov 01 22:30:51.401752 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:51.401503 5011 scope.go:122] "RemoveContainer" containerID="170d46e91f9bd9e30ff744c342939c658c3044ba07f6d7c3f8d52892faae84b8" Nov 01 22:30:51.409966 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:51.409947 5011 scope.go:122] "RemoveContainer" containerID="170d46e91f9bd9e30ff744c342939c658c3044ba07f6d7c3f8d52892faae84b8" Nov 01 22:30:51.410250 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:30:51.410232 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"170d46e91f9bd9e30ff744c342939c658c3044ba07f6d7c3f8d52892faae84b8\": container with ID starting with 170d46e91f9bd9e30ff744c342939c658c3044ba07f6d7c3f8d52892faae84b8 not found: ID does not exist" containerID="170d46e91f9bd9e30ff744c342939c658c3044ba07f6d7c3f8d52892faae84b8" Nov 01 22:30:51.410308 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:51.410257 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"170d46e91f9bd9e30ff744c342939c658c3044ba07f6d7c3f8d52892faae84b8"} err="failed to get container status \"170d46e91f9bd9e30ff744c342939c658c3044ba07f6d7c3f8d52892faae84b8\": rpc error: code = NotFound desc = could not find container \"170d46e91f9bd9e30ff744c342939c658c3044ba07f6d7c3f8d52892faae84b8\": container with ID starting with 170d46e91f9bd9e30ff744c342939c658c3044ba07f6d7c3f8d52892faae84b8 not found: ID does not exist" Nov 01 22:30:51.421039 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:51.421016 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rwmtq"] Nov 01 22:30:51.422017 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:51.421997 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rwmtq"] Nov 01 22:30:51.743879 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:51.743823 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddb4396c-e40c-4257-b10a-87f228f3d7e9" path="/var/lib/kubelet/pods/ddb4396c-e40c-4257-b10a-87f228f3d7e9/volumes" Nov 01 22:30:59.393679 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:30:59.393651 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-54nss" Nov 01 22:31:16.958296 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:31:16.958254 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:31:28.168867 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.167872 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-9ns67"] Nov 01 22:31:28.168867 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.168562 5011 cpu_manager.go:414] "containerMap: removing container" podUID="ddb4396c-e40c-4257-b10a-87f228f3d7e9" containerName="authorino" Nov 01 22:31:28.168867 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.168579 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddb4396c-e40c-4257-b10a-87f228f3d7e9" containerName="authorino" Nov 01 22:31:28.168867 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.168775 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddb4396c-e40c-4257-b10a-87f228f3d7e9" containerName="authorino" Nov 01 22:31:28.176417 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.176385 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-9ns67" Nov 01 22:31:28.179986 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.179958 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-9ns67"] Nov 01 22:31:28.182176 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.182154 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 01 22:31:28.182327 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.182241 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 01 22:31:28.182433 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.182359 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 01 22:31:28.182433 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.182413 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 01 22:31:28.182433 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.182429 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 01 22:31:28.182676 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.182661 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ztmdx\"" Nov 01 22:31:28.266521 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.266483 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb564eda-deb4-46a2-9688-7a15e683bfcc-metrics-certs\") pod \"maas-controller-579cf7c6d-9ns67\" (UID: \"eb564eda-deb4-46a2-9688-7a15e683bfcc\") " pod="opendatahub/maas-controller-579cf7c6d-9ns67" Nov 01 22:31:28.266521 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.266521 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb564eda-deb4-46a2-9688-7a15e683bfcc-webhook-cert\") pod \"maas-controller-579cf7c6d-9ns67\" (UID: \"eb564eda-deb4-46a2-9688-7a15e683bfcc\") " pod="opendatahub/maas-controller-579cf7c6d-9ns67" Nov 01 22:31:28.266756 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.266642 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vknxc\" (UniqueName: \"kubernetes.io/projected/eb564eda-deb4-46a2-9688-7a15e683bfcc-kube-api-access-vknxc\") pod \"maas-controller-579cf7c6d-9ns67\" (UID: \"eb564eda-deb4-46a2-9688-7a15e683bfcc\") " pod="opendatahub/maas-controller-579cf7c6d-9ns67" Nov 01 22:31:28.367261 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.367219 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb564eda-deb4-46a2-9688-7a15e683bfcc-metrics-certs\") pod \"maas-controller-579cf7c6d-9ns67\" (UID: \"eb564eda-deb4-46a2-9688-7a15e683bfcc\") " pod="opendatahub/maas-controller-579cf7c6d-9ns67" Nov 01 22:31:28.367261 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.367259 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb564eda-deb4-46a2-9688-7a15e683bfcc-webhook-cert\") pod \"maas-controller-579cf7c6d-9ns67\" (UID: \"eb564eda-deb4-46a2-9688-7a15e683bfcc\") " pod="opendatahub/maas-controller-579cf7c6d-9ns67" Nov 01 22:31:28.367477 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.367309 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vknxc\" (UniqueName: \"kubernetes.io/projected/eb564eda-deb4-46a2-9688-7a15e683bfcc-kube-api-access-vknxc\") pod \"maas-controller-579cf7c6d-9ns67\" (UID: \"eb564eda-deb4-46a2-9688-7a15e683bfcc\") " pod="opendatahub/maas-controller-579cf7c6d-9ns67" Nov 01 22:31:28.369989 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.369960 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb564eda-deb4-46a2-9688-7a15e683bfcc-webhook-cert\") pod \"maas-controller-579cf7c6d-9ns67\" (UID: \"eb564eda-deb4-46a2-9688-7a15e683bfcc\") " pod="opendatahub/maas-controller-579cf7c6d-9ns67" Nov 01 22:31:28.370265 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.370006 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb564eda-deb4-46a2-9688-7a15e683bfcc-metrics-certs\") pod \"maas-controller-579cf7c6d-9ns67\" (UID: \"eb564eda-deb4-46a2-9688-7a15e683bfcc\") " pod="opendatahub/maas-controller-579cf7c6d-9ns67" Nov 01 22:31:28.380043 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.380020 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vknxc\" (UniqueName: \"kubernetes.io/projected/eb564eda-deb4-46a2-9688-7a15e683bfcc-kube-api-access-vknxc\") pod \"maas-controller-579cf7c6d-9ns67\" (UID: \"eb564eda-deb4-46a2-9688-7a15e683bfcc\") " pod="opendatahub/maas-controller-579cf7c6d-9ns67" Nov 01 22:31:28.483181 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.483152 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-66565fcd54-xfn5s"] Nov 01 22:31:28.485960 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.485944 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66565fcd54-xfn5s" Nov 01 22:31:28.492347 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.492323 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-9ns67" Nov 01 22:31:28.497343 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.495064 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66565fcd54-xfn5s"] Nov 01 22:31:28.569513 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.569479 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8b5k\" (UniqueName: \"kubernetes.io/projected/4f6e51ef-8558-4cc7-9c47-77dba02aadc8-kube-api-access-z8b5k\") pod \"maas-controller-66565fcd54-xfn5s\" (UID: \"4f6e51ef-8558-4cc7-9c47-77dba02aadc8\") " pod="opendatahub/maas-controller-66565fcd54-xfn5s" Nov 01 22:31:28.569650 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.569543 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4f6e51ef-8558-4cc7-9c47-77dba02aadc8-webhook-cert\") pod \"maas-controller-66565fcd54-xfn5s\" (UID: \"4f6e51ef-8558-4cc7-9c47-77dba02aadc8\") " pod="opendatahub/maas-controller-66565fcd54-xfn5s" Nov 01 22:31:28.569650 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.569623 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f6e51ef-8558-4cc7-9c47-77dba02aadc8-metrics-certs\") pod \"maas-controller-66565fcd54-xfn5s\" (UID: \"4f6e51ef-8558-4cc7-9c47-77dba02aadc8\") " pod="opendatahub/maas-controller-66565fcd54-xfn5s" Nov 01 22:31:28.671017 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.670984 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8b5k\" (UniqueName: \"kubernetes.io/projected/4f6e51ef-8558-4cc7-9c47-77dba02aadc8-kube-api-access-z8b5k\") pod \"maas-controller-66565fcd54-xfn5s\" (UID: \"4f6e51ef-8558-4cc7-9c47-77dba02aadc8\") " pod="opendatahub/maas-controller-66565fcd54-xfn5s" Nov 01 22:31:28.671181 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.671028 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4f6e51ef-8558-4cc7-9c47-77dba02aadc8-webhook-cert\") pod \"maas-controller-66565fcd54-xfn5s\" (UID: \"4f6e51ef-8558-4cc7-9c47-77dba02aadc8\") " pod="opendatahub/maas-controller-66565fcd54-xfn5s" Nov 01 22:31:28.671181 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.671051 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f6e51ef-8558-4cc7-9c47-77dba02aadc8-metrics-certs\") pod \"maas-controller-66565fcd54-xfn5s\" (UID: \"4f6e51ef-8558-4cc7-9c47-77dba02aadc8\") " pod="opendatahub/maas-controller-66565fcd54-xfn5s" Nov 01 22:31:28.673557 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.673532 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4f6e51ef-8558-4cc7-9c47-77dba02aadc8-webhook-cert\") pod \"maas-controller-66565fcd54-xfn5s\" (UID: \"4f6e51ef-8558-4cc7-9c47-77dba02aadc8\") " pod="opendatahub/maas-controller-66565fcd54-xfn5s" Nov 01 22:31:28.673557 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.673555 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f6e51ef-8558-4cc7-9c47-77dba02aadc8-metrics-certs\") pod \"maas-controller-66565fcd54-xfn5s\" (UID: \"4f6e51ef-8558-4cc7-9c47-77dba02aadc8\") " pod="opendatahub/maas-controller-66565fcd54-xfn5s" Nov 01 22:31:28.679384 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.679363 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8b5k\" (UniqueName: \"kubernetes.io/projected/4f6e51ef-8558-4cc7-9c47-77dba02aadc8-kube-api-access-z8b5k\") pod \"maas-controller-66565fcd54-xfn5s\" (UID: \"4f6e51ef-8558-4cc7-9c47-77dba02aadc8\") " pod="opendatahub/maas-controller-66565fcd54-xfn5s" Nov 01 22:31:28.766192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.766167 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-9ns67"] Nov 01 22:31:28.768556 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:31:28.768531 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb564eda_deb4_46a2_9688_7a15e683bfcc.slice/crio-f7cba3f6ff8958e678647c7dbd31e9d2285485473509ec78095fd00b66a0dc0e WatchSource:0}: Error finding container f7cba3f6ff8958e678647c7dbd31e9d2285485473509ec78095fd00b66a0dc0e: Status 404 returned error can't find the container with id f7cba3f6ff8958e678647c7dbd31e9d2285485473509ec78095fd00b66a0dc0e Nov 01 22:31:28.799336 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:28.799314 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66565fcd54-xfn5s" Nov 01 22:31:29.099610 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:29.099534 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66565fcd54-xfn5s"] Nov 01 22:31:29.102532 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:31:29.102491 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f6e51ef_8558_4cc7_9c47_77dba02aadc8.slice/crio-a5dcd2d55443fc4ea8cd29c4dd081f2f4337d105a2c9d79486bfcf7689803f90 WatchSource:0}: Error finding container a5dcd2d55443fc4ea8cd29c4dd081f2f4337d105a2c9d79486bfcf7689803f90: Status 404 returned error can't find the container with id a5dcd2d55443fc4ea8cd29c4dd081f2f4337d105a2c9d79486bfcf7689803f90 Nov 01 22:31:29.566507 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:29.566470 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66565fcd54-xfn5s" event={"ID":"4f6e51ef-8558-4cc7-9c47-77dba02aadc8","Type":"ContainerStarted","Data":"a5dcd2d55443fc4ea8cd29c4dd081f2f4337d105a2c9d79486bfcf7689803f90"} Nov 01 22:31:29.567686 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:29.567663 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-9ns67" event={"ID":"eb564eda-deb4-46a2-9688-7a15e683bfcc","Type":"ContainerStarted","Data":"f7cba3f6ff8958e678647c7dbd31e9d2285485473509ec78095fd00b66a0dc0e"} Nov 01 22:31:32.587645 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:32.587605 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66565fcd54-xfn5s" event={"ID":"4f6e51ef-8558-4cc7-9c47-77dba02aadc8","Type":"ContainerStarted","Data":"60547e47f7252507a1a21ea8fe8a32eb170a3859bc0818ddb93d0301cae5ec8b"} Nov 01 22:31:32.588124 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:32.587775 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-66565fcd54-xfn5s" Nov 01 22:31:32.589328 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:32.589304 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-9ns67" event={"ID":"eb564eda-deb4-46a2-9688-7a15e683bfcc","Type":"ContainerStarted","Data":"11b32ca04e2a87410c1c66b5377c0970ef214a3614caecc04fcbfb515b8d1954"} Nov 01 22:31:32.589453 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:32.589434 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-9ns67" Nov 01 22:31:32.613680 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:32.613627 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-66565fcd54-xfn5s" podStartSLOduration=1.813611751 podStartE2EDuration="4.613612289s" totalImagesPullingTime="2.800000538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:32.612217733 +0000 UTC m=+461.016808366" watchObservedRunningTime="2026-11-01 22:31:32.613612289 +0000 UTC m=+461.018202921" Nov 01 22:31:32.638928 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:32.638875 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-9ns67" podStartSLOduration=1.4273531099999999 podStartE2EDuration="4.638860116s" totalImagesPullingTime="3.211507006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:32.637562817 +0000 UTC m=+461.042153449" watchObservedRunningTime="2026-11-01 22:31:32.638860116 +0000 UTC m=+461.043450748" Nov 01 22:31:39.575540 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:39.575501 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-gf7xg"] Nov 01 22:31:39.578843 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:39.578813 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-gf7xg" Nov 01 22:31:39.596490 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:39.596460 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-6k42j\"" Nov 01 22:31:39.597150 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:39.597134 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 01 22:31:39.602149 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:39.602121 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-gf7xg"] Nov 01 22:31:39.670527 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:39.670496 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nvpf\" (UniqueName: \"kubernetes.io/projected/946fc3be-df64-414f-908d-c1d85fceb06f-kube-api-access-9nvpf\") pod \"kserve-controller-manager-776cc66d47-gf7xg\" (UID: \"946fc3be-df64-414f-908d-c1d85fceb06f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-gf7xg" Nov 01 22:31:39.670705 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:39.670583 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/946fc3be-df64-414f-908d-c1d85fceb06f-cert\") pod \"kserve-controller-manager-776cc66d47-gf7xg\" (UID: \"946fc3be-df64-414f-908d-c1d85fceb06f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-gf7xg" Nov 01 22:31:39.771387 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:39.771350 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nvpf\" (UniqueName: \"kubernetes.io/projected/946fc3be-df64-414f-908d-c1d85fceb06f-kube-api-access-9nvpf\") pod \"kserve-controller-manager-776cc66d47-gf7xg\" (UID: \"946fc3be-df64-414f-908d-c1d85fceb06f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-gf7xg" Nov 01 22:31:39.771577 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:39.771513 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/946fc3be-df64-414f-908d-c1d85fceb06f-cert\") pod \"kserve-controller-manager-776cc66d47-gf7xg\" (UID: \"946fc3be-df64-414f-908d-c1d85fceb06f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-gf7xg" Nov 01 22:31:39.774354 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:39.774323 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/946fc3be-df64-414f-908d-c1d85fceb06f-cert\") pod \"kserve-controller-manager-776cc66d47-gf7xg\" (UID: \"946fc3be-df64-414f-908d-c1d85fceb06f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-gf7xg" Nov 01 22:31:39.788653 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:39.788627 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nvpf\" (UniqueName: \"kubernetes.io/projected/946fc3be-df64-414f-908d-c1d85fceb06f-kube-api-access-9nvpf\") pod \"kserve-controller-manager-776cc66d47-gf7xg\" (UID: \"946fc3be-df64-414f-908d-c1d85fceb06f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-gf7xg" Nov 01 22:31:39.892097 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:39.892020 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-gf7xg" Nov 01 22:31:40.180321 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:40.180299 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-gf7xg"] Nov 01 22:31:40.182678 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:31:40.182649 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946fc3be_df64_414f_908d_c1d85fceb06f.slice/crio-bfa8351dd51607cc53c69ed103779321ed1ecaad7f73262c8d8a242881fb9b54 WatchSource:0}: Error finding container bfa8351dd51607cc53c69ed103779321ed1ecaad7f73262c8d8a242881fb9b54: Status 404 returned error can't find the container with id bfa8351dd51607cc53c69ed103779321ed1ecaad7f73262c8d8a242881fb9b54 Nov 01 22:31:40.625937 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:40.625818 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-gf7xg" event={"ID":"946fc3be-df64-414f-908d-c1d85fceb06f","Type":"ContainerStarted","Data":"bfa8351dd51607cc53c69ed103779321ed1ecaad7f73262c8d8a242881fb9b54"} Nov 01 22:31:40.708825 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:40.708785 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvvdl"] Nov 01 22:31:40.713364 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:40.713332 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvvdl" Nov 01 22:31:40.729333 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:40.729307 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvvdl"] Nov 01 22:31:40.780307 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:40.780273 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c511deaf-bfd4-4c7f-a12e-39f8b80a3a61-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvvdl\" (UID: \"c511deaf-bfd4-4c7f-a12e-39f8b80a3a61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvvdl" Nov 01 22:31:40.780467 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:40.780328 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c511deaf-bfd4-4c7f-a12e-39f8b80a3a61-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvvdl\" (UID: \"c511deaf-bfd4-4c7f-a12e-39f8b80a3a61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvvdl" Nov 01 22:31:40.780467 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:40.780426 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c511deaf-bfd4-4c7f-a12e-39f8b80a3a61-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvvdl\" (UID: \"c511deaf-bfd4-4c7f-a12e-39f8b80a3a61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvvdl" Nov 01 22:31:40.780467 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:40.780458 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c511deaf-bfd4-4c7f-a12e-39f8b80a3a61-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvvdl\" (UID: \"c511deaf-bfd4-4c7f-a12e-39f8b80a3a61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvvdl" Nov 01 22:31:40.780609 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:40.780487 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t6r7\" (UniqueName: \"kubernetes.io/projected/c511deaf-bfd4-4c7f-a12e-39f8b80a3a61-kube-api-access-6t6r7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvvdl\" (UID: \"c511deaf-bfd4-4c7f-a12e-39f8b80a3a61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvvdl" Nov 01 22:31:40.780609 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:40.780505 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c511deaf-bfd4-4c7f-a12e-39f8b80a3a61-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvvdl\" (UID: \"c511deaf-bfd4-4c7f-a12e-39f8b80a3a61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvvdl" Nov 01 22:31:40.780691 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:40.780595 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c511deaf-bfd4-4c7f-a12e-39f8b80a3a61-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvvdl\" (UID: \"c511deaf-bfd4-4c7f-a12e-39f8b80a3a61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvvdl" Nov 01 22:31:40.881171 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:40.881087 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c511deaf-bfd4-4c7f-a12e-39f8b80a3a61-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvvdl\" (UID: \"c511deaf-bfd4-4c7f-a12e-39f8b80a3a61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvvdl" Nov 01 22:31:40.881171 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:40.881126 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c511deaf-bfd4-4c7f-a12e-39f8b80a3a61-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvvdl\" (UID: \"c511deaf-bfd4-4c7f-a12e-39f8b80a3a61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvvdl" Nov 01 22:31:40.881393 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:40.881254 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t6r7\" (UniqueName: \"kubernetes.io/projected/c511deaf-bfd4-4c7f-a12e-39f8b80a3a61-kube-api-access-6t6r7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvvdl\" (UID: \"c511deaf-bfd4-4c7f-a12e-39f8b80a3a61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvvdl" Nov 01 22:31:40.881393 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:40.881291 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c511deaf-bfd4-4c7f-a12e-39f8b80a3a61-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvvdl\" (UID: \"c511deaf-bfd4-4c7f-a12e-39f8b80a3a61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvvdl" Nov 01 22:31:40.881393 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:40.881310 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c511deaf-bfd4-4c7f-a12e-39f8b80a3a61-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvvdl\" (UID: \"c511deaf-bfd4-4c7f-a12e-39f8b80a3a61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvvdl" Nov 01 22:31:40.881393 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:40.881373 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c511deaf-bfd4-4c7f-a12e-39f8b80a3a61-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvvdl\" (UID: \"c511deaf-bfd4-4c7f-a12e-39f8b80a3a61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvvdl" Nov 01 22:31:40.881581 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:40.881439 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c511deaf-bfd4-4c7f-a12e-39f8b80a3a61-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvvdl\" (UID: \"c511deaf-bfd4-4c7f-a12e-39f8b80a3a61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvvdl" Nov 01 22:31:40.881988 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:40.881918 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c511deaf-bfd4-4c7f-a12e-39f8b80a3a61-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvvdl\" (UID: \"c511deaf-bfd4-4c7f-a12e-39f8b80a3a61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvvdl" Nov 01 22:31:40.883550 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:40.883527 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c511deaf-bfd4-4c7f-a12e-39f8b80a3a61-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvvdl\" (UID: \"c511deaf-bfd4-4c7f-a12e-39f8b80a3a61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvvdl" Nov 01 22:31:40.883760 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:40.883742 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c511deaf-bfd4-4c7f-a12e-39f8b80a3a61-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvvdl\" (UID: \"c511deaf-bfd4-4c7f-a12e-39f8b80a3a61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvvdl" Nov 01 22:31:40.884203 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:40.884188 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c511deaf-bfd4-4c7f-a12e-39f8b80a3a61-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvvdl\" (UID: \"c511deaf-bfd4-4c7f-a12e-39f8b80a3a61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvvdl" Nov 01 22:31:40.884726 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:40.884706 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c511deaf-bfd4-4c7f-a12e-39f8b80a3a61-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvvdl\" (UID: \"c511deaf-bfd4-4c7f-a12e-39f8b80a3a61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvvdl" Nov 01 22:31:40.890634 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:40.890610 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t6r7\" (UniqueName: \"kubernetes.io/projected/c511deaf-bfd4-4c7f-a12e-39f8b80a3a61-kube-api-access-6t6r7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvvdl\" (UID: \"c511deaf-bfd4-4c7f-a12e-39f8b80a3a61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvvdl" Nov 01 22:31:40.893938 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:40.893920 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c511deaf-bfd4-4c7f-a12e-39f8b80a3a61-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mvvdl\" (UID: \"c511deaf-bfd4-4c7f-a12e-39f8b80a3a61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvvdl" Nov 01 22:31:41.026380 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:41.026343 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvvdl" Nov 01 22:31:41.349507 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:41.349440 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvvdl"] Nov 01 22:31:41.352436 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:31:41.352393 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc511deaf_bfd4_4c7f_a12e_39f8b80a3a61.slice/crio-6fb13c2e09ecfdc73e0fe83333bcf98b490ff186b4a5f6db1d6b6f1221c01e59 WatchSource:0}: Error finding container 6fb13c2e09ecfdc73e0fe83333bcf98b490ff186b4a5f6db1d6b6f1221c01e59: Status 404 returned error can't find the container with id 6fb13c2e09ecfdc73e0fe83333bcf98b490ff186b4a5f6db1d6b6f1221c01e59 Nov 01 22:31:41.355014 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:41.354978 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:31:41.355170 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:41.355055 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:31:41.633041 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:41.632953 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvvdl" event={"ID":"c511deaf-bfd4-4c7f-a12e-39f8b80a3a61","Type":"ContainerStarted","Data":"d3dd7eba59a319c3b0887044ca27b73d4f688bfd821827b28b667f90785eb976"} Nov 01 22:31:41.633041 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:41.633006 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvvdl" event={"ID":"c511deaf-bfd4-4c7f-a12e-39f8b80a3a61","Type":"ContainerStarted","Data":"6fb13c2e09ecfdc73e0fe83333bcf98b490ff186b4a5f6db1d6b6f1221c01e59"} Nov 01 22:31:41.633527 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:41.633140 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvvdl" Nov 01 22:31:41.655179 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:41.655108 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvvdl" podStartSLOduration=1.655095249 podStartE2EDuration="1.655095249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:41.654267044 +0000 UTC m=+470.058857678" watchObservedRunningTime="2026-11-01 22:31:41.655095249 +0000 UTC m=+470.059685880" Nov 01 22:31:42.645170 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:42.645137 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-gf7xg" event={"ID":"946fc3be-df64-414f-908d-c1d85fceb06f","Type":"ContainerStarted","Data":"7834e7c4b2b973a259a69c44f3b3f9f5c15b02e47e1e83effb881febbd3034e6"} Nov 01 22:31:42.646998 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:42.646956 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mvvdl" Nov 01 22:31:42.666584 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:42.666530 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-gf7xg" podStartSLOduration=1.278845767 podStartE2EDuration="3.666516996s" totalImagesPullingTime="2.387671229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:42.664909024 +0000 UTC m=+471.069499660" watchObservedRunningTime="2026-11-01 22:31:42.666516996 +0000 UTC m=+471.071107663" Nov 01 22:31:42.726660 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:42.726623 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96"] Nov 01 22:31:42.726915 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:42.726874 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" podUID="571cb0bc-4b05-4d60-968d-44b609698f40" containerName="discovery" containerID="cri-o://56fa7bd9471ba623ae9e6b45557aeb35409b8389fbe8c7cabcf055c6580e2338" gracePeriod=30 Nov 01 22:31:43.002298 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.002271 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" Nov 01 22:31:43.102993 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.102910 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/571cb0bc-4b05-4d60-968d-44b609698f40-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/571cb0bc-4b05-4d60-968d-44b609698f40-istio-csr-ca-configmap\") pod \"571cb0bc-4b05-4d60-968d-44b609698f40\" (UID: \"571cb0bc-4b05-4d60-968d-44b609698f40\") " Nov 01 22:31:43.102993 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.102945 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/571cb0bc-4b05-4d60-968d-44b609698f40-cacerts\" (UniqueName: \"kubernetes.io/secret/571cb0bc-4b05-4d60-968d-44b609698f40-cacerts\") pod \"571cb0bc-4b05-4d60-968d-44b609698f40\" (UID: \"571cb0bc-4b05-4d60-968d-44b609698f40\") " Nov 01 22:31:43.102993 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.102974 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/571cb0bc-4b05-4d60-968d-44b609698f40-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/571cb0bc-4b05-4d60-968d-44b609698f40-local-certs\") pod \"571cb0bc-4b05-4d60-968d-44b609698f40\" (UID: \"571cb0bc-4b05-4d60-968d-44b609698f40\") " Nov 01 22:31:43.103271 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.103010 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/571cb0bc-4b05-4d60-968d-44b609698f40-kube-api-access-jqp4s\" (UniqueName: \"kubernetes.io/projected/571cb0bc-4b05-4d60-968d-44b609698f40-kube-api-access-jqp4s\") pod \"571cb0bc-4b05-4d60-968d-44b609698f40\" (UID: \"571cb0bc-4b05-4d60-968d-44b609698f40\") " Nov 01 22:31:43.103271 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.103136 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/571cb0bc-4b05-4d60-968d-44b609698f40-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/571cb0bc-4b05-4d60-968d-44b609698f40-istio-kubeconfig\") pod \"571cb0bc-4b05-4d60-968d-44b609698f40\" (UID: \"571cb0bc-4b05-4d60-968d-44b609698f40\") " Nov 01 22:31:43.103271 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.103194 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/571cb0bc-4b05-4d60-968d-44b609698f40-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/571cb0bc-4b05-4d60-968d-44b609698f40-istio-csr-dns-cert\") pod \"571cb0bc-4b05-4d60-968d-44b609698f40\" (UID: \"571cb0bc-4b05-4d60-968d-44b609698f40\") " Nov 01 22:31:43.103271 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.103238 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/571cb0bc-4b05-4d60-968d-44b609698f40-istio-token\" (UniqueName: \"kubernetes.io/projected/571cb0bc-4b05-4d60-968d-44b609698f40-istio-token\") pod \"571cb0bc-4b05-4d60-968d-44b609698f40\" (UID: \"571cb0bc-4b05-4d60-968d-44b609698f40\") " Nov 01 22:31:43.103481 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.103428 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/571cb0bc-4b05-4d60-968d-44b609698f40-istio-csr-ca-configmap" pod "571cb0bc-4b05-4d60-968d-44b609698f40" (UID: "571cb0bc-4b05-4d60-968d-44b609698f40"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:31:43.103574 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.103557 5011 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/571cb0bc-4b05-4d60-968d-44b609698f40-istio-csr-ca-configmap\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:43.105735 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.105700 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/571cb0bc-4b05-4d60-968d-44b609698f40-cacerts" pod "571cb0bc-4b05-4d60-968d-44b609698f40" (UID: "571cb0bc-4b05-4d60-968d-44b609698f40"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:31:43.105898 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.105732 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/571cb0bc-4b05-4d60-968d-44b609698f40-istio-kubeconfig" pod "571cb0bc-4b05-4d60-968d-44b609698f40" (UID: "571cb0bc-4b05-4d60-968d-44b609698f40"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:31:43.105977 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.105934 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/571cb0bc-4b05-4d60-968d-44b609698f40-local-certs" pod "571cb0bc-4b05-4d60-968d-44b609698f40" (UID: "571cb0bc-4b05-4d60-968d-44b609698f40"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:31:43.105977 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.105939 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/571cb0bc-4b05-4d60-968d-44b609698f40-istio-csr-dns-cert" pod "571cb0bc-4b05-4d60-968d-44b609698f40" (UID: "571cb0bc-4b05-4d60-968d-44b609698f40"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:31:43.106101 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.106078 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/571cb0bc-4b05-4d60-968d-44b609698f40-istio-token" pod "571cb0bc-4b05-4d60-968d-44b609698f40" (UID: "571cb0bc-4b05-4d60-968d-44b609698f40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:43.106349 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.106332 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/571cb0bc-4b05-4d60-968d-44b609698f40-kube-api-access-jqp4s" pod "571cb0bc-4b05-4d60-968d-44b609698f40" (UID: "571cb0bc-4b05-4d60-968d-44b609698f40"). InnerVolumeSpecName "kube-api-access-jqp4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:43.204467 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.204436 5011 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/571cb0bc-4b05-4d60-968d-44b609698f40-cacerts\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:43.204467 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.204463 5011 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/571cb0bc-4b05-4d60-968d-44b609698f40-local-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:43.204467 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.204474 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqp4s\" (UniqueName: \"kubernetes.io/projected/571cb0bc-4b05-4d60-968d-44b609698f40-kube-api-access-jqp4s\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:43.204679 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.204484 5011 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/571cb0bc-4b05-4d60-968d-44b609698f40-istio-kubeconfig\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:43.204679 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.204499 5011 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/571cb0bc-4b05-4d60-968d-44b609698f40-istio-csr-dns-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:43.204679 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.204507 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/571cb0bc-4b05-4d60-968d-44b609698f40-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:43.595852 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.595798 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-66565fcd54-xfn5s" Nov 01 22:31:43.596043 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.595980 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-9ns67" Nov 01 22:31:43.651730 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.651694 5011 generic.go:356] "Generic (PLEG): container finished" podID="571cb0bc-4b05-4d60-968d-44b609698f40" containerID="56fa7bd9471ba623ae9e6b45557aeb35409b8389fbe8c7cabcf055c6580e2338" exitCode=0 Nov 01 22:31:43.652357 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.651741 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" Nov 01 22:31:43.652357 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.651787 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" event={"ID":"571cb0bc-4b05-4d60-968d-44b609698f40","Type":"ContainerDied","Data":"56fa7bd9471ba623ae9e6b45557aeb35409b8389fbe8c7cabcf055c6580e2338"} Nov 01 22:31:43.652357 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.651856 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" event={"ID":"571cb0bc-4b05-4d60-968d-44b609698f40","Type":"ContainerDied","Data":"cae92c7ddc2dc9d4c83226cd2353f4553deb6c6270e76610805a0e4a0f95f452"} Nov 01 22:31:43.652357 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.651877 5011 scope.go:122] "RemoveContainer" containerID="56fa7bd9471ba623ae9e6b45557aeb35409b8389fbe8c7cabcf055c6580e2338" Nov 01 22:31:43.653758 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.653710 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-gf7xg" Nov 01 22:31:43.662543 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.662524 5011 scope.go:122] "RemoveContainer" containerID="56fa7bd9471ba623ae9e6b45557aeb35409b8389fbe8c7cabcf055c6580e2338" Nov 01 22:31:43.662927 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:31:43.662900 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56fa7bd9471ba623ae9e6b45557aeb35409b8389fbe8c7cabcf055c6580e2338\": container with ID starting with 56fa7bd9471ba623ae9e6b45557aeb35409b8389fbe8c7cabcf055c6580e2338 not found: ID does not exist" containerID="56fa7bd9471ba623ae9e6b45557aeb35409b8389fbe8c7cabcf055c6580e2338" Nov 01 22:31:43.663016 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.662939 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56fa7bd9471ba623ae9e6b45557aeb35409b8389fbe8c7cabcf055c6580e2338"} err="failed to get container status \"56fa7bd9471ba623ae9e6b45557aeb35409b8389fbe8c7cabcf055c6580e2338\": rpc error: code = NotFound desc = could not find container \"56fa7bd9471ba623ae9e6b45557aeb35409b8389fbe8c7cabcf055c6580e2338\": container with ID starting with 56fa7bd9471ba623ae9e6b45557aeb35409b8389fbe8c7cabcf055c6580e2338 not found: ID does not exist" Nov 01 22:31:43.673373 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.673342 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-9ns67"] Nov 01 22:31:43.673590 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.673559 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-9ns67" podUID="eb564eda-deb4-46a2-9688-7a15e683bfcc" containerName="manager" containerID="cri-o://11b32ca04e2a87410c1c66b5377c0970ef214a3614caecc04fcbfb515b8d1954" gracePeriod=30 Nov 01 22:31:43.700496 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.700466 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96"] Nov 01 22:31:43.707909 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.707886 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96"] Nov 01 22:31:43.745605 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.745565 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="571cb0bc-4b05-4d60-968d-44b609698f40" path="/var/lib/kubelet/pods/571cb0bc-4b05-4d60-968d-44b609698f40/volumes" Nov 01 22:31:43.908789 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:43.908764 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-9ns67" Nov 01 22:31:44.012765 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:44.012738 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb564eda-deb4-46a2-9688-7a15e683bfcc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb564eda-deb4-46a2-9688-7a15e683bfcc-metrics-certs\") pod \"eb564eda-deb4-46a2-9688-7a15e683bfcc\" (UID: \"eb564eda-deb4-46a2-9688-7a15e683bfcc\") " Nov 01 22:31:44.012962 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:44.012784 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb564eda-deb4-46a2-9688-7a15e683bfcc-kube-api-access-vknxc\" (UniqueName: \"kubernetes.io/projected/eb564eda-deb4-46a2-9688-7a15e683bfcc-kube-api-access-vknxc\") pod \"eb564eda-deb4-46a2-9688-7a15e683bfcc\" (UID: \"eb564eda-deb4-46a2-9688-7a15e683bfcc\") " Nov 01 22:31:44.012962 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:44.012898 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb564eda-deb4-46a2-9688-7a15e683bfcc-webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb564eda-deb4-46a2-9688-7a15e683bfcc-webhook-cert\") pod \"eb564eda-deb4-46a2-9688-7a15e683bfcc\" (UID: \"eb564eda-deb4-46a2-9688-7a15e683bfcc\") " Nov 01 22:31:44.015164 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:44.015124 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb564eda-deb4-46a2-9688-7a15e683bfcc-metrics-certs" pod "eb564eda-deb4-46a2-9688-7a15e683bfcc" (UID: "eb564eda-deb4-46a2-9688-7a15e683bfcc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:31:44.015291 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:44.015231 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb564eda-deb4-46a2-9688-7a15e683bfcc-webhook-cert" pod "eb564eda-deb4-46a2-9688-7a15e683bfcc" (UID: "eb564eda-deb4-46a2-9688-7a15e683bfcc"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:31:44.015384 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:44.015360 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb564eda-deb4-46a2-9688-7a15e683bfcc-kube-api-access-vknxc" pod "eb564eda-deb4-46a2-9688-7a15e683bfcc" (UID: "eb564eda-deb4-46a2-9688-7a15e683bfcc"). InnerVolumeSpecName "kube-api-access-vknxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:44.114167 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:44.114130 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vknxc\" (UniqueName: \"kubernetes.io/projected/eb564eda-deb4-46a2-9688-7a15e683bfcc-kube-api-access-vknxc\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:44.114167 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:44.114158 5011 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb564eda-deb4-46a2-9688-7a15e683bfcc-webhook-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:44.114167 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:44.114167 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb564eda-deb4-46a2-9688-7a15e683bfcc-metrics-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:44.658311 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:44.658276 5011 generic.go:356] "Generic (PLEG): container finished" podID="eb564eda-deb4-46a2-9688-7a15e683bfcc" containerID="11b32ca04e2a87410c1c66b5377c0970ef214a3614caecc04fcbfb515b8d1954" exitCode=0 Nov 01 22:31:44.658768 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:44.658325 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-9ns67" Nov 01 22:31:44.658768 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:44.658354 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-9ns67" event={"ID":"eb564eda-deb4-46a2-9688-7a15e683bfcc","Type":"ContainerDied","Data":"11b32ca04e2a87410c1c66b5377c0970ef214a3614caecc04fcbfb515b8d1954"} Nov 01 22:31:44.658768 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:44.658403 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-9ns67" event={"ID":"eb564eda-deb4-46a2-9688-7a15e683bfcc","Type":"ContainerDied","Data":"f7cba3f6ff8958e678647c7dbd31e9d2285485473509ec78095fd00b66a0dc0e"} Nov 01 22:31:44.658768 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:44.658426 5011 scope.go:122] "RemoveContainer" containerID="11b32ca04e2a87410c1c66b5377c0970ef214a3614caecc04fcbfb515b8d1954" Nov 01 22:31:44.667275 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:44.667260 5011 scope.go:122] "RemoveContainer" containerID="11b32ca04e2a87410c1c66b5377c0970ef214a3614caecc04fcbfb515b8d1954" Nov 01 22:31:44.667519 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:31:44.667499 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11b32ca04e2a87410c1c66b5377c0970ef214a3614caecc04fcbfb515b8d1954\": container with ID starting with 11b32ca04e2a87410c1c66b5377c0970ef214a3614caecc04fcbfb515b8d1954 not found: ID does not exist" containerID="11b32ca04e2a87410c1c66b5377c0970ef214a3614caecc04fcbfb515b8d1954" Nov 01 22:31:44.667555 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:44.667532 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11b32ca04e2a87410c1c66b5377c0970ef214a3614caecc04fcbfb515b8d1954"} err="failed to get container status \"11b32ca04e2a87410c1c66b5377c0970ef214a3614caecc04fcbfb515b8d1954\": rpc error: code = NotFound desc = could not find container \"11b32ca04e2a87410c1c66b5377c0970ef214a3614caecc04fcbfb515b8d1954\": container with ID starting with 11b32ca04e2a87410c1c66b5377c0970ef214a3614caecc04fcbfb515b8d1954 not found: ID does not exist" Nov 01 22:31:44.681980 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:44.681949 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-9ns67"] Nov 01 22:31:44.685423 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:44.685403 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-9ns67"] Nov 01 22:31:45.744757 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:45.744719 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb564eda-deb4-46a2-9688-7a15e683bfcc" path="/var/lib/kubelet/pods/eb564eda-deb4-46a2-9688-7a15e683bfcc/volumes" Nov 01 22:31:46.959084 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:31:46.959041 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:31:47.987527 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:47.987484 5011 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-96w96 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.23:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:31:47.987921 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:31:47.987549 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-96w96" podUID="571cb0bc-4b05-4d60-968d-44b609698f40" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.23:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:32:04.892685 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:04.892652 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-626tr"] Nov 01 22:32:04.893304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:04.893167 5011 cpu_manager.go:414] "containerMap: removing container" podUID="eb564eda-deb4-46a2-9688-7a15e683bfcc" containerName="manager" Nov 01 22:32:04.893304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:04.893188 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb564eda-deb4-46a2-9688-7a15e683bfcc" containerName="manager" Nov 01 22:32:04.893304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:04.893213 5011 cpu_manager.go:414] "containerMap: removing container" podUID="571cb0bc-4b05-4d60-968d-44b609698f40" containerName="discovery" Nov 01 22:32:04.893304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:04.893222 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="571cb0bc-4b05-4d60-968d-44b609698f40" containerName="discovery" Nov 01 22:32:04.893304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:04.893295 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb564eda-deb4-46a2-9688-7a15e683bfcc" containerName="manager" Nov 01 22:32:04.893304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:04.893307 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="571cb0bc-4b05-4d60-968d-44b609698f40" containerName="discovery" Nov 01 22:32:04.899978 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:04.899951 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-626tr" Nov 01 22:32:04.912153 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:04.911869 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 22:32:04.912153 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:04.911870 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-57phm\"" Nov 01 22:32:04.912153 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:04.911909 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 01 22:32:04.913030 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:04.913009 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:32:04.913147 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:04.913012 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:32:04.915049 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:04.915026 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-626tr"] Nov 01 22:32:04.999159 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:04.999125 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bb2c77a-f912-4862-a11a-3ec993c0a63e-metrics-certs\") pod \"maas-api-6d669c7fcd-626tr\" (UID: \"8bb2c77a-f912-4862-a11a-3ec993c0a63e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-626tr" Nov 01 22:32:04.999355 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:04.999187 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bb2c77a-f912-4862-a11a-3ec993c0a63e-maas-api-tls\") pod \"maas-api-6d669c7fcd-626tr\" (UID: \"8bb2c77a-f912-4862-a11a-3ec993c0a63e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-626tr" Nov 01 22:32:04.999355 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:04.999275 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bb2c77a-f912-4862-a11a-3ec993c0a63e-openshift-service-ca\") pod \"maas-api-6d669c7fcd-626tr\" (UID: \"8bb2c77a-f912-4862-a11a-3ec993c0a63e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-626tr" Nov 01 22:32:04.999355 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:04.999317 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t2qk\" (UniqueName: \"kubernetes.io/projected/8bb2c77a-f912-4862-a11a-3ec993c0a63e-kube-api-access-5t2qk\") pod \"maas-api-6d669c7fcd-626tr\" (UID: \"8bb2c77a-f912-4862-a11a-3ec993c0a63e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-626tr" Nov 01 22:32:05.099919 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:05.099881 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bb2c77a-f912-4862-a11a-3ec993c0a63e-metrics-certs\") pod \"maas-api-6d669c7fcd-626tr\" (UID: \"8bb2c77a-f912-4862-a11a-3ec993c0a63e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-626tr" Nov 01 22:32:05.100111 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:05.099942 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bb2c77a-f912-4862-a11a-3ec993c0a63e-maas-api-tls\") pod \"maas-api-6d669c7fcd-626tr\" (UID: \"8bb2c77a-f912-4862-a11a-3ec993c0a63e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-626tr" Nov 01 22:32:05.100111 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:05.100072 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bb2c77a-f912-4862-a11a-3ec993c0a63e-openshift-service-ca\") pod \"maas-api-6d669c7fcd-626tr\" (UID: \"8bb2c77a-f912-4862-a11a-3ec993c0a63e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-626tr" Nov 01 22:32:05.100239 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:05.100115 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t2qk\" (UniqueName: \"kubernetes.io/projected/8bb2c77a-f912-4862-a11a-3ec993c0a63e-kube-api-access-5t2qk\") pod \"maas-api-6d669c7fcd-626tr\" (UID: \"8bb2c77a-f912-4862-a11a-3ec993c0a63e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-626tr" Nov 01 22:32:05.100621 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:05.100601 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bb2c77a-f912-4862-a11a-3ec993c0a63e-openshift-service-ca\") pod \"maas-api-6d669c7fcd-626tr\" (UID: \"8bb2c77a-f912-4862-a11a-3ec993c0a63e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-626tr" Nov 01 22:32:05.102417 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:05.102397 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bb2c77a-f912-4862-a11a-3ec993c0a63e-maas-api-tls\") pod \"maas-api-6d669c7fcd-626tr\" (UID: \"8bb2c77a-f912-4862-a11a-3ec993c0a63e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-626tr" Nov 01 22:32:05.102516 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:05.102460 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bb2c77a-f912-4862-a11a-3ec993c0a63e-metrics-certs\") pod \"maas-api-6d669c7fcd-626tr\" (UID: \"8bb2c77a-f912-4862-a11a-3ec993c0a63e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-626tr" Nov 01 22:32:05.109848 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:05.109206 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t2qk\" (UniqueName: \"kubernetes.io/projected/8bb2c77a-f912-4862-a11a-3ec993c0a63e-kube-api-access-5t2qk\") pod \"maas-api-6d669c7fcd-626tr\" (UID: \"8bb2c77a-f912-4862-a11a-3ec993c0a63e\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-626tr" Nov 01 22:32:05.213525 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:05.213484 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-626tr" Nov 01 22:32:05.487216 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:05.487194 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-626tr"] Nov 01 22:32:05.489729 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:32:05.489698 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bb2c77a_f912_4862_a11a_3ec993c0a63e.slice/crio-9179bb63cedd22c682f3e5b8f9c21ea73d034fbbe3242874d0ee30e1450bb1c6 WatchSource:0}: Error finding container 9179bb63cedd22c682f3e5b8f9c21ea73d034fbbe3242874d0ee30e1450bb1c6: Status 404 returned error can't find the container with id 9179bb63cedd22c682f3e5b8f9c21ea73d034fbbe3242874d0ee30e1450bb1c6 Nov 01 22:32:05.661608 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:05.661568 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:32:05.755728 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:05.755642 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-626tr" event={"ID":"8bb2c77a-f912-4862-a11a-3ec993c0a63e","Type":"ContainerStarted","Data":"9179bb63cedd22c682f3e5b8f9c21ea73d034fbbe3242874d0ee30e1450bb1c6"} Nov 01 22:32:07.766281 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:07.766245 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-626tr" event={"ID":"8bb2c77a-f912-4862-a11a-3ec993c0a63e","Type":"ContainerStarted","Data":"3ce70830832ea0809ba8cc8536134a15581615501eadbb5f48a63df20f9fca02"} Nov 01 22:32:07.766897 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:07.766318 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-626tr" Nov 01 22:32:07.783824 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:07.783776 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-626tr" podStartSLOduration=2.214914933 podStartE2EDuration="3.78376276s" totalImagesPullingTime="1.568847827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:32:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:32:07.783185068 +0000 UTC m=+496.187775701" watchObservedRunningTime="2026-11-01 22:32:07.78376276 +0000 UTC m=+496.188353371" Nov 01 22:32:13.776348 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:13.776319 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-626tr" Nov 01 22:32:14.664203 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:14.664175 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-gf7xg" Nov 01 22:32:15.444664 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.444632 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-nld84"] Nov 01 22:32:15.449771 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.449751 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-nld84" Nov 01 22:32:15.456172 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.456143 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-nld84"] Nov 01 22:32:15.478592 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.478549 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22qgz\" (UniqueName: \"kubernetes.io/projected/4dfe364b-0593-489b-97be-bcb93fe4e391-kube-api-access-22qgz\") pod \"authorino-6f6944bcdd-nld84\" (UID: \"4dfe364b-0593-489b-97be-bcb93fe4e391\") " pod="kuadrant-system/authorino-6f6944bcdd-nld84" Nov 01 22:32:15.506806 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.506771 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-nld84"] Nov 01 22:32:15.507019 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:32:15.506999 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-22qgz], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-nld84" podUID="4dfe364b-0593-489b-97be-bcb93fe4e391" Nov 01 22:32:15.533235 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.533198 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-qgk6b"] Nov 01 22:32:15.537692 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.537667 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-qgk6b" Nov 01 22:32:15.543252 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.543226 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 01 22:32:15.544398 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.544374 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-qgk6b"] Nov 01 22:32:15.579614 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.579580 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f566z\" (UniqueName: \"kubernetes.io/projected/c726855b-339d-4a56-95ea-aa96c586a721-kube-api-access-f566z\") pod \"authorino-6889cbd8c4-qgk6b\" (UID: \"c726855b-339d-4a56-95ea-aa96c586a721\") " pod="kuadrant-system/authorino-6889cbd8c4-qgk6b" Nov 01 22:32:15.579803 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.579622 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22qgz\" (UniqueName: \"kubernetes.io/projected/4dfe364b-0593-489b-97be-bcb93fe4e391-kube-api-access-22qgz\") pod \"authorino-6f6944bcdd-nld84\" (UID: \"4dfe364b-0593-489b-97be-bcb93fe4e391\") " pod="kuadrant-system/authorino-6f6944bcdd-nld84" Nov 01 22:32:15.579803 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.579649 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c726855b-339d-4a56-95ea-aa96c586a721-tls-cert\") pod \"authorino-6889cbd8c4-qgk6b\" (UID: \"c726855b-339d-4a56-95ea-aa96c586a721\") " pod="kuadrant-system/authorino-6889cbd8c4-qgk6b" Nov 01 22:32:15.595441 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.595406 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22qgz\" (UniqueName: \"kubernetes.io/projected/4dfe364b-0593-489b-97be-bcb93fe4e391-kube-api-access-22qgz\") pod \"authorino-6f6944bcdd-nld84\" (UID: \"4dfe364b-0593-489b-97be-bcb93fe4e391\") " pod="kuadrant-system/authorino-6f6944bcdd-nld84" Nov 01 22:32:15.680944 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.680912 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f566z\" (UniqueName: \"kubernetes.io/projected/c726855b-339d-4a56-95ea-aa96c586a721-kube-api-access-f566z\") pod \"authorino-6889cbd8c4-qgk6b\" (UID: \"c726855b-339d-4a56-95ea-aa96c586a721\") " pod="kuadrant-system/authorino-6889cbd8c4-qgk6b" Nov 01 22:32:15.681121 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.680950 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c726855b-339d-4a56-95ea-aa96c586a721-tls-cert\") pod \"authorino-6889cbd8c4-qgk6b\" (UID: \"c726855b-339d-4a56-95ea-aa96c586a721\") " pod="kuadrant-system/authorino-6889cbd8c4-qgk6b" Nov 01 22:32:15.683494 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.683467 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c726855b-339d-4a56-95ea-aa96c586a721-tls-cert\") pod \"authorino-6889cbd8c4-qgk6b\" (UID: \"c726855b-339d-4a56-95ea-aa96c586a721\") " pod="kuadrant-system/authorino-6889cbd8c4-qgk6b" Nov 01 22:32:15.689255 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.689220 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f566z\" (UniqueName: \"kubernetes.io/projected/c726855b-339d-4a56-95ea-aa96c586a721-kube-api-access-f566z\") pod \"authorino-6889cbd8c4-qgk6b\" (UID: \"c726855b-339d-4a56-95ea-aa96c586a721\") " pod="kuadrant-system/authorino-6889cbd8c4-qgk6b" Nov 01 22:32:15.764449 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.764366 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-qgk6b"] Nov 01 22:32:15.764593 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.764562 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-qgk6b" Nov 01 22:32:15.791000 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.790973 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bb7596679-xvxvn"] Nov 01 22:32:15.795540 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.795511 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bb7596679-xvxvn" Nov 01 22:32:15.799846 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.799804 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-nld84" Nov 01 22:32:15.804629 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.804605 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bb7596679-xvxvn"] Nov 01 22:32:15.812702 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.812676 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-nld84" Nov 01 22:32:15.882415 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.882385 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dfe364b-0593-489b-97be-bcb93fe4e391-kube-api-access-22qgz\" (UniqueName: \"kubernetes.io/projected/4dfe364b-0593-489b-97be-bcb93fe4e391-kube-api-access-22qgz\") pod \"4dfe364b-0593-489b-97be-bcb93fe4e391\" (UID: \"4dfe364b-0593-489b-97be-bcb93fe4e391\") " Nov 01 22:32:15.882554 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.882534 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d23bde01-3662-4d8c-8581-afcd8d7f74e1-tls-cert\") pod \"authorino-7bb7596679-xvxvn\" (UID: \"d23bde01-3662-4d8c-8581-afcd8d7f74e1\") " pod="kuadrant-system/authorino-7bb7596679-xvxvn" Nov 01 22:32:15.882689 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.882669 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzwsm\" (UniqueName: \"kubernetes.io/projected/d23bde01-3662-4d8c-8581-afcd8d7f74e1-kube-api-access-dzwsm\") pod \"authorino-7bb7596679-xvxvn\" (UID: \"d23bde01-3662-4d8c-8581-afcd8d7f74e1\") " pod="kuadrant-system/authorino-7bb7596679-xvxvn" Nov 01 22:32:15.884552 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.884532 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dfe364b-0593-489b-97be-bcb93fe4e391-kube-api-access-22qgz" pod "4dfe364b-0593-489b-97be-bcb93fe4e391" (UID: "4dfe364b-0593-489b-97be-bcb93fe4e391"). InnerVolumeSpecName "kube-api-access-22qgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:15.983739 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.983705 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzwsm\" (UniqueName: \"kubernetes.io/projected/d23bde01-3662-4d8c-8581-afcd8d7f74e1-kube-api-access-dzwsm\") pod \"authorino-7bb7596679-xvxvn\" (UID: \"d23bde01-3662-4d8c-8581-afcd8d7f74e1\") " pod="kuadrant-system/authorino-7bb7596679-xvxvn" Nov 01 22:32:15.983956 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.983768 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d23bde01-3662-4d8c-8581-afcd8d7f74e1-tls-cert\") pod \"authorino-7bb7596679-xvxvn\" (UID: \"d23bde01-3662-4d8c-8581-afcd8d7f74e1\") " pod="kuadrant-system/authorino-7bb7596679-xvxvn" Nov 01 22:32:15.983956 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.983803 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22qgz\" (UniqueName: \"kubernetes.io/projected/4dfe364b-0593-489b-97be-bcb93fe4e391-kube-api-access-22qgz\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:15.986252 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.986229 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d23bde01-3662-4d8c-8581-afcd8d7f74e1-tls-cert\") pod \"authorino-7bb7596679-xvxvn\" (UID: \"d23bde01-3662-4d8c-8581-afcd8d7f74e1\") " pod="kuadrant-system/authorino-7bb7596679-xvxvn" Nov 01 22:32:15.991864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:15.991823 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzwsm\" (UniqueName: \"kubernetes.io/projected/d23bde01-3662-4d8c-8581-afcd8d7f74e1-kube-api-access-dzwsm\") pod \"authorino-7bb7596679-xvxvn\" (UID: \"d23bde01-3662-4d8c-8581-afcd8d7f74e1\") " pod="kuadrant-system/authorino-7bb7596679-xvxvn" Nov 01 22:32:16.041707 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:16.041682 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-qgk6b"] Nov 01 22:32:16.044289 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:32:16.044261 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc726855b_339d_4a56_95ea_aa96c586a721.slice/crio-ae7b86a7d7854776dcb424263c2192dcdb770071cc74cdd327d11be164bb06e3 WatchSource:0}: Error finding container ae7b86a7d7854776dcb424263c2192dcdb770071cc74cdd327d11be164bb06e3: Status 404 returned error can't find the container with id ae7b86a7d7854776dcb424263c2192dcdb770071cc74cdd327d11be164bb06e3 Nov 01 22:32:16.112518 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:16.112482 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bb7596679-xvxvn" Nov 01 22:32:16.386450 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:16.386424 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bb7596679-xvxvn"] Nov 01 22:32:16.388402 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:32:16.388373 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd23bde01_3662_4d8c_8581_afcd8d7f74e1.slice/crio-a5a53a05859b7d9189bc8d69d8d5c78170aaf6b19d7be0a1ea9e1605bd8c7a32 WatchSource:0}: Error finding container a5a53a05859b7d9189bc8d69d8d5c78170aaf6b19d7be0a1ea9e1605bd8c7a32: Status 404 returned error can't find the container with id a5a53a05859b7d9189bc8d69d8d5c78170aaf6b19d7be0a1ea9e1605bd8c7a32 Nov 01 22:32:16.804873 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:16.804818 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bb7596679-xvxvn" event={"ID":"d23bde01-3662-4d8c-8581-afcd8d7f74e1","Type":"ContainerStarted","Data":"a5a53a05859b7d9189bc8d69d8d5c78170aaf6b19d7be0a1ea9e1605bd8c7a32"} Nov 01 22:32:16.806042 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:16.806017 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-qgk6b" event={"ID":"c726855b-339d-4a56-95ea-aa96c586a721","Type":"ContainerStarted","Data":"ae7b86a7d7854776dcb424263c2192dcdb770071cc74cdd327d11be164bb06e3"} Nov 01 22:32:16.806163 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:16.806028 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-nld84" Nov 01 22:32:16.835026 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:16.834999 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-nld84"] Nov 01 22:32:16.840758 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:16.840730 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-nld84"] Nov 01 22:32:16.957951 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:32:16.957914 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:32:17.747708 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:17.747675 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dfe364b-0593-489b-97be-bcb93fe4e391" path="/var/lib/kubelet/pods/4dfe364b-0593-489b-97be-bcb93fe4e391/volumes" Nov 01 22:32:17.814222 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:17.814181 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bb7596679-xvxvn" event={"ID":"d23bde01-3662-4d8c-8581-afcd8d7f74e1","Type":"ContainerStarted","Data":"5edb8b05546ee4bfd22cb2c021f2dab243ff66446f8761b60965a9e0fd6cfc2c"} Nov 01 22:32:17.816464 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:17.816425 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-qgk6b" event={"ID":"c726855b-339d-4a56-95ea-aa96c586a721","Type":"ContainerStarted","Data":"9dcb9c1acaa9089c6364f30afacd01491cb8cae556554995c761dbe9c0c55c4a"} Nov 01 22:32:17.816601 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:17.816490 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-qgk6b" podUID="c726855b-339d-4a56-95ea-aa96c586a721" containerName="authorino" containerID="cri-o://9dcb9c1acaa9089c6364f30afacd01491cb8cae556554995c761dbe9c0c55c4a" gracePeriod=30 Nov 01 22:32:17.831401 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:17.831323 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7bb7596679-xvxvn" podStartSLOduration=2.147675248 podStartE2EDuration="2.831302515s" totalImagesPullingTime="683.627267ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:32:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:32:17.829894144 +0000 UTC m=+506.234484775" watchObservedRunningTime="2026-11-01 22:32:17.831302515 +0000 UTC m=+506.235893147" Nov 01 22:32:17.851697 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:17.851640 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-qgk6b" podStartSLOduration=1.819467409 podStartE2EDuration="2.851619432s" totalImagesPullingTime="1.032152023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:32:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:32:17.849760773 +0000 UTC m=+506.254351417" watchObservedRunningTime="2026-11-01 22:32:17.851619432 +0000 UTC m=+506.256210066" Nov 01 22:32:17.857089 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:17.857064 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9d2z2"] Nov 01 22:32:17.857257 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:17.857239 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-9d2z2" podUID="956c392a-258c-4c1c-aada-74d72921ecc2" containerName="authorino" containerID="cri-o://559b97c81510dc9d8d8647cbd2ccaefc31b17a70fd091076329fdcae97ce7934" gracePeriod=30 Nov 01 22:32:18.114774 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:18.114750 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-qgk6b" Nov 01 22:32:18.117880 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:18.117854 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-9d2z2" Nov 01 22:32:18.209527 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:18.209495 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c726855b-339d-4a56-95ea-aa96c586a721-kube-api-access-f566z\" (UniqueName: \"kubernetes.io/projected/c726855b-339d-4a56-95ea-aa96c586a721-kube-api-access-f566z\") pod \"c726855b-339d-4a56-95ea-aa96c586a721\" (UID: \"c726855b-339d-4a56-95ea-aa96c586a721\") " Nov 01 22:32:18.209698 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:18.209560 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/956c392a-258c-4c1c-aada-74d72921ecc2-kube-api-access-n9hgf\" (UniqueName: \"kubernetes.io/projected/956c392a-258c-4c1c-aada-74d72921ecc2-kube-api-access-n9hgf\") pod \"956c392a-258c-4c1c-aada-74d72921ecc2\" (UID: \"956c392a-258c-4c1c-aada-74d72921ecc2\") " Nov 01 22:32:18.209698 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:18.209586 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c726855b-339d-4a56-95ea-aa96c586a721-tls-cert\" (UniqueName: \"kubernetes.io/secret/c726855b-339d-4a56-95ea-aa96c586a721-tls-cert\") pod \"c726855b-339d-4a56-95ea-aa96c586a721\" (UID: \"c726855b-339d-4a56-95ea-aa96c586a721\") " Nov 01 22:32:18.211913 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:18.211874 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c726855b-339d-4a56-95ea-aa96c586a721-kube-api-access-f566z" pod "c726855b-339d-4a56-95ea-aa96c586a721" (UID: "c726855b-339d-4a56-95ea-aa96c586a721"). InnerVolumeSpecName "kube-api-access-f566z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:18.211913 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:18.211884 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/956c392a-258c-4c1c-aada-74d72921ecc2-kube-api-access-n9hgf" pod "956c392a-258c-4c1c-aada-74d72921ecc2" (UID: "956c392a-258c-4c1c-aada-74d72921ecc2"). InnerVolumeSpecName "kube-api-access-n9hgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:18.219691 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:18.219659 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c726855b-339d-4a56-95ea-aa96c586a721-tls-cert" pod "c726855b-339d-4a56-95ea-aa96c586a721" (UID: "c726855b-339d-4a56-95ea-aa96c586a721"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:32:18.310412 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:18.310329 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9hgf\" (UniqueName: \"kubernetes.io/projected/956c392a-258c-4c1c-aada-74d72921ecc2-kube-api-access-n9hgf\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:18.310412 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:18.310356 5011 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c726855b-339d-4a56-95ea-aa96c586a721-tls-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:18.310412 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:18.310367 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f566z\" (UniqueName: \"kubernetes.io/projected/c726855b-339d-4a56-95ea-aa96c586a721-kube-api-access-f566z\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:18.822713 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:18.822674 5011 generic.go:356] "Generic (PLEG): container finished" podID="956c392a-258c-4c1c-aada-74d72921ecc2" containerID="559b97c81510dc9d8d8647cbd2ccaefc31b17a70fd091076329fdcae97ce7934" exitCode=0 Nov 01 22:32:18.823232 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:18.822721 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-9d2z2" Nov 01 22:32:18.823232 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:18.822754 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-9d2z2" event={"ID":"956c392a-258c-4c1c-aada-74d72921ecc2","Type":"ContainerDied","Data":"559b97c81510dc9d8d8647cbd2ccaefc31b17a70fd091076329fdcae97ce7934"} Nov 01 22:32:18.823232 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:18.822811 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-9d2z2" event={"ID":"956c392a-258c-4c1c-aada-74d72921ecc2","Type":"ContainerDied","Data":"dbda4b69480002b4e1dffe5f09b8a72e6a043d93bf33ba7db3f928e219fec28c"} Nov 01 22:32:18.823232 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:18.822853 5011 scope.go:122] "RemoveContainer" containerID="559b97c81510dc9d8d8647cbd2ccaefc31b17a70fd091076329fdcae97ce7934" Nov 01 22:32:18.824201 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:18.824175 5011 generic.go:356] "Generic (PLEG): container finished" podID="c726855b-339d-4a56-95ea-aa96c586a721" containerID="9dcb9c1acaa9089c6364f30afacd01491cb8cae556554995c761dbe9c0c55c4a" exitCode=0 Nov 01 22:32:18.824327 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:18.824229 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-qgk6b" Nov 01 22:32:18.824327 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:18.824282 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-qgk6b" event={"ID":"c726855b-339d-4a56-95ea-aa96c586a721","Type":"ContainerDied","Data":"9dcb9c1acaa9089c6364f30afacd01491cb8cae556554995c761dbe9c0c55c4a"} Nov 01 22:32:18.824327 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:18.824312 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-qgk6b" event={"ID":"c726855b-339d-4a56-95ea-aa96c586a721","Type":"ContainerDied","Data":"ae7b86a7d7854776dcb424263c2192dcdb770071cc74cdd327d11be164bb06e3"} Nov 01 22:32:18.831306 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:18.831278 5011 scope.go:122] "RemoveContainer" containerID="559b97c81510dc9d8d8647cbd2ccaefc31b17a70fd091076329fdcae97ce7934" Nov 01 22:32:18.831582 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:32:18.831564 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"559b97c81510dc9d8d8647cbd2ccaefc31b17a70fd091076329fdcae97ce7934\": container with ID starting with 559b97c81510dc9d8d8647cbd2ccaefc31b17a70fd091076329fdcae97ce7934 not found: ID does not exist" containerID="559b97c81510dc9d8d8647cbd2ccaefc31b17a70fd091076329fdcae97ce7934" Nov 01 22:32:18.831644 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:18.831592 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"559b97c81510dc9d8d8647cbd2ccaefc31b17a70fd091076329fdcae97ce7934"} err="failed to get container status \"559b97c81510dc9d8d8647cbd2ccaefc31b17a70fd091076329fdcae97ce7934\": rpc error: code = NotFound desc = could not find container \"559b97c81510dc9d8d8647cbd2ccaefc31b17a70fd091076329fdcae97ce7934\": container with ID starting with 559b97c81510dc9d8d8647cbd2ccaefc31b17a70fd091076329fdcae97ce7934 not found: ID does not exist" Nov 01 22:32:18.831644 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:18.831612 5011 scope.go:122] "RemoveContainer" containerID="9dcb9c1acaa9089c6364f30afacd01491cb8cae556554995c761dbe9c0c55c4a" Nov 01 22:32:18.840713 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:18.840699 5011 scope.go:122] "RemoveContainer" containerID="9dcb9c1acaa9089c6364f30afacd01491cb8cae556554995c761dbe9c0c55c4a" Nov 01 22:32:18.840995 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:32:18.840978 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dcb9c1acaa9089c6364f30afacd01491cb8cae556554995c761dbe9c0c55c4a\": container with ID starting with 9dcb9c1acaa9089c6364f30afacd01491cb8cae556554995c761dbe9c0c55c4a not found: ID does not exist" containerID="9dcb9c1acaa9089c6364f30afacd01491cb8cae556554995c761dbe9c0c55c4a" Nov 01 22:32:18.841065 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:18.841006 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dcb9c1acaa9089c6364f30afacd01491cb8cae556554995c761dbe9c0c55c4a"} err="failed to get container status \"9dcb9c1acaa9089c6364f30afacd01491cb8cae556554995c761dbe9c0c55c4a\": rpc error: code = NotFound desc = could not find container \"9dcb9c1acaa9089c6364f30afacd01491cb8cae556554995c761dbe9c0c55c4a\": container with ID starting with 9dcb9c1acaa9089c6364f30afacd01491cb8cae556554995c761dbe9c0c55c4a not found: ID does not exist" Nov 01 22:32:18.851277 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:18.851255 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-qgk6b"] Nov 01 22:32:18.855179 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:18.855159 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-qgk6b"] Nov 01 22:32:18.864297 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:18.864275 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9d2z2"] Nov 01 22:32:18.868327 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:18.868308 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9d2z2"] Nov 01 22:32:19.744846 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:19.744807 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="956c392a-258c-4c1c-aada-74d72921ecc2" path="/var/lib/kubelet/pods/956c392a-258c-4c1c-aada-74d72921ecc2/volumes" Nov 01 22:32:19.745192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:19.745177 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c726855b-339d-4a56-95ea-aa96c586a721" path="/var/lib/kubelet/pods/c726855b-339d-4a56-95ea-aa96c586a721/volumes" Nov 01 22:32:26.967273 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:26.967235 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-626tr"] Nov 01 22:32:26.967661 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:26.967437 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-626tr" podUID="8bb2c77a-f912-4862-a11a-3ec993c0a63e" containerName="maas-api" containerID="cri-o://3ce70830832ea0809ba8cc8536134a15581615501eadbb5f48a63df20f9fca02" gracePeriod=30 Nov 01 22:32:27.215205 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:27.215173 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-626tr" Nov 01 22:32:27.292544 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:27.292441 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bb2c77a-f912-4862-a11a-3ec993c0a63e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bb2c77a-f912-4862-a11a-3ec993c0a63e-maas-api-tls\") pod \"8bb2c77a-f912-4862-a11a-3ec993c0a63e\" (UID: \"8bb2c77a-f912-4862-a11a-3ec993c0a63e\") " Nov 01 22:32:27.292544 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:27.292512 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bb2c77a-f912-4862-a11a-3ec993c0a63e-kube-api-access-5t2qk\" (UniqueName: \"kubernetes.io/projected/8bb2c77a-f912-4862-a11a-3ec993c0a63e-kube-api-access-5t2qk\") pod \"8bb2c77a-f912-4862-a11a-3ec993c0a63e\" (UID: \"8bb2c77a-f912-4862-a11a-3ec993c0a63e\") " Nov 01 22:32:27.292751 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:27.292620 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bb2c77a-f912-4862-a11a-3ec993c0a63e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bb2c77a-f912-4862-a11a-3ec993c0a63e-metrics-certs\") pod \"8bb2c77a-f912-4862-a11a-3ec993c0a63e\" (UID: \"8bb2c77a-f912-4862-a11a-3ec993c0a63e\") " Nov 01 22:32:27.292751 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:27.292670 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bb2c77a-f912-4862-a11a-3ec993c0a63e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bb2c77a-f912-4862-a11a-3ec993c0a63e-openshift-service-ca\") pod \"8bb2c77a-f912-4862-a11a-3ec993c0a63e\" (UID: \"8bb2c77a-f912-4862-a11a-3ec993c0a63e\") " Nov 01 22:32:27.293094 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:27.293068 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bb2c77a-f912-4862-a11a-3ec993c0a63e-openshift-service-ca" pod "8bb2c77a-f912-4862-a11a-3ec993c0a63e" (UID: "8bb2c77a-f912-4862-a11a-3ec993c0a63e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:32:27.295065 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:27.295038 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb2c77a-f912-4862-a11a-3ec993c0a63e-metrics-certs" pod "8bb2c77a-f912-4862-a11a-3ec993c0a63e" (UID: "8bb2c77a-f912-4862-a11a-3ec993c0a63e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:32:27.295291 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:27.295269 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb2c77a-f912-4862-a11a-3ec993c0a63e-kube-api-access-5t2qk" pod "8bb2c77a-f912-4862-a11a-3ec993c0a63e" (UID: "8bb2c77a-f912-4862-a11a-3ec993c0a63e"). InnerVolumeSpecName "kube-api-access-5t2qk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:32:27.295291 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:27.295274 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb2c77a-f912-4862-a11a-3ec993c0a63e-maas-api-tls" pod "8bb2c77a-f912-4862-a11a-3ec993c0a63e" (UID: "8bb2c77a-f912-4862-a11a-3ec993c0a63e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:32:27.393953 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:27.393922 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t2qk\" (UniqueName: \"kubernetes.io/projected/8bb2c77a-f912-4862-a11a-3ec993c0a63e-kube-api-access-5t2qk\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:27.393953 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:27.393952 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bb2c77a-f912-4862-a11a-3ec993c0a63e-metrics-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:27.393953 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:27.393962 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bb2c77a-f912-4862-a11a-3ec993c0a63e-openshift-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:27.394208 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:27.393972 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bb2c77a-f912-4862-a11a-3ec993c0a63e-maas-api-tls\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:32:27.866268 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:27.866235 5011 generic.go:356] "Generic (PLEG): container finished" podID="8bb2c77a-f912-4862-a11a-3ec993c0a63e" containerID="3ce70830832ea0809ba8cc8536134a15581615501eadbb5f48a63df20f9fca02" exitCode=0 Nov 01 22:32:27.866436 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:27.866298 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-626tr" Nov 01 22:32:27.866436 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:27.866323 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-626tr" event={"ID":"8bb2c77a-f912-4862-a11a-3ec993c0a63e","Type":"ContainerDied","Data":"3ce70830832ea0809ba8cc8536134a15581615501eadbb5f48a63df20f9fca02"} Nov 01 22:32:27.866436 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:27.866399 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-626tr" event={"ID":"8bb2c77a-f912-4862-a11a-3ec993c0a63e","Type":"ContainerDied","Data":"9179bb63cedd22c682f3e5b8f9c21ea73d034fbbe3242874d0ee30e1450bb1c6"} Nov 01 22:32:27.866538 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:27.866446 5011 scope.go:122] "RemoveContainer" containerID="3ce70830832ea0809ba8cc8536134a15581615501eadbb5f48a63df20f9fca02" Nov 01 22:32:27.877694 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:27.876500 5011 scope.go:122] "RemoveContainer" containerID="3ce70830832ea0809ba8cc8536134a15581615501eadbb5f48a63df20f9fca02" Nov 01 22:32:27.878006 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:32:27.877983 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ce70830832ea0809ba8cc8536134a15581615501eadbb5f48a63df20f9fca02\": container with ID starting with 3ce70830832ea0809ba8cc8536134a15581615501eadbb5f48a63df20f9fca02 not found: ID does not exist" containerID="3ce70830832ea0809ba8cc8536134a15581615501eadbb5f48a63df20f9fca02" Nov 01 22:32:27.878097 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:27.878015 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ce70830832ea0809ba8cc8536134a15581615501eadbb5f48a63df20f9fca02"} err="failed to get container status \"3ce70830832ea0809ba8cc8536134a15581615501eadbb5f48a63df20f9fca02\": rpc error: code = NotFound desc = could not find container \"3ce70830832ea0809ba8cc8536134a15581615501eadbb5f48a63df20f9fca02\": container with ID starting with 3ce70830832ea0809ba8cc8536134a15581615501eadbb5f48a63df20f9fca02 not found: ID does not exist" Nov 01 22:32:27.882231 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:27.882198 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-626tr"] Nov 01 22:32:27.884477 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:27.884449 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-626tr"] Nov 01 22:32:29.744870 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:32:29.744816 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bb2c77a-f912-4862-a11a-3ec993c0a63e" path="/var/lib/kubelet/pods/8bb2c77a-f912-4862-a11a-3ec993c0a63e/volumes" Nov 01 22:32:46.959087 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:32:46.958981 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:33:16.958787 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:33:16.958749 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:33:30.119901 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.119862 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-7q99w"] Nov 01 22:33:30.120334 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.120190 5011 cpu_manager.go:414] "containerMap: removing container" podUID="8bb2c77a-f912-4862-a11a-3ec993c0a63e" containerName="maas-api" Nov 01 22:33:30.120334 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.120204 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bb2c77a-f912-4862-a11a-3ec993c0a63e" containerName="maas-api" Nov 01 22:33:30.120334 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.120220 5011 cpu_manager.go:414] "containerMap: removing container" podUID="956c392a-258c-4c1c-aada-74d72921ecc2" containerName="authorino" Nov 01 22:33:30.120334 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.120225 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="956c392a-258c-4c1c-aada-74d72921ecc2" containerName="authorino" Nov 01 22:33:30.120334 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.120248 5011 cpu_manager.go:414] "containerMap: removing container" podUID="c726855b-339d-4a56-95ea-aa96c586a721" containerName="authorino" Nov 01 22:33:30.120334 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.120253 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c726855b-339d-4a56-95ea-aa96c586a721" containerName="authorino" Nov 01 22:33:30.120334 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.120304 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="c726855b-339d-4a56-95ea-aa96c586a721" containerName="authorino" Nov 01 22:33:30.120334 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.120312 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bb2c77a-f912-4862-a11a-3ec993c0a63e" containerName="maas-api" Nov 01 22:33:30.120334 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.120318 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="956c392a-258c-4c1c-aada-74d72921ecc2" containerName="authorino" Nov 01 22:33:30.123483 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.123458 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7q99w" Nov 01 22:33:30.129129 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.129091 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 01 22:33:30.129559 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.129538 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 01 22:33:30.129686 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.129657 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-njcj2\"" Nov 01 22:33:30.129686 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.129658 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 01 22:33:30.136276 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.136250 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-7q99w"] Nov 01 22:33:30.232730 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.232697 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/643613a9-01d4-4d8e-bfa3-3cc8b7e118b3-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7q99w\" (UID: \"643613a9-01d4-4d8e-bfa3-3cc8b7e118b3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7q99w" Nov 01 22:33:30.232730 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.232736 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/643613a9-01d4-4d8e-bfa3-3cc8b7e118b3-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7q99w\" (UID: \"643613a9-01d4-4d8e-bfa3-3cc8b7e118b3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7q99w" Nov 01 22:33:30.232981 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.232753 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/643613a9-01d4-4d8e-bfa3-3cc8b7e118b3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7q99w\" (UID: \"643613a9-01d4-4d8e-bfa3-3cc8b7e118b3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7q99w" Nov 01 22:33:30.232981 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.232771 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-499fs\" (UniqueName: \"kubernetes.io/projected/643613a9-01d4-4d8e-bfa3-3cc8b7e118b3-kube-api-access-499fs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7q99w\" (UID: \"643613a9-01d4-4d8e-bfa3-3cc8b7e118b3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7q99w" Nov 01 22:33:30.232981 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.232900 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/643613a9-01d4-4d8e-bfa3-3cc8b7e118b3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7q99w\" (UID: \"643613a9-01d4-4d8e-bfa3-3cc8b7e118b3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7q99w" Nov 01 22:33:30.232981 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.232918 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/643613a9-01d4-4d8e-bfa3-3cc8b7e118b3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7q99w\" (UID: \"643613a9-01d4-4d8e-bfa3-3cc8b7e118b3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7q99w" Nov 01 22:33:30.333691 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.333654 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/643613a9-01d4-4d8e-bfa3-3cc8b7e118b3-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7q99w\" (UID: \"643613a9-01d4-4d8e-bfa3-3cc8b7e118b3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7q99w" Nov 01 22:33:30.333691 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.333691 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/643613a9-01d4-4d8e-bfa3-3cc8b7e118b3-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7q99w\" (UID: \"643613a9-01d4-4d8e-bfa3-3cc8b7e118b3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7q99w" Nov 01 22:33:30.333971 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.333714 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/643613a9-01d4-4d8e-bfa3-3cc8b7e118b3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7q99w\" (UID: \"643613a9-01d4-4d8e-bfa3-3cc8b7e118b3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7q99w" Nov 01 22:33:30.333971 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.333737 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-499fs\" (UniqueName: \"kubernetes.io/projected/643613a9-01d4-4d8e-bfa3-3cc8b7e118b3-kube-api-access-499fs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7q99w\" (UID: \"643613a9-01d4-4d8e-bfa3-3cc8b7e118b3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7q99w" Nov 01 22:33:30.333971 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.333769 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/643613a9-01d4-4d8e-bfa3-3cc8b7e118b3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7q99w\" (UID: \"643613a9-01d4-4d8e-bfa3-3cc8b7e118b3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7q99w" Nov 01 22:33:30.333971 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.333867 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/643613a9-01d4-4d8e-bfa3-3cc8b7e118b3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7q99w\" (UID: \"643613a9-01d4-4d8e-bfa3-3cc8b7e118b3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7q99w" Nov 01 22:33:30.334175 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.334113 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/643613a9-01d4-4d8e-bfa3-3cc8b7e118b3-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7q99w\" (UID: \"643613a9-01d4-4d8e-bfa3-3cc8b7e118b3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7q99w" Nov 01 22:33:30.334216 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.334177 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/643613a9-01d4-4d8e-bfa3-3cc8b7e118b3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7q99w\" (UID: \"643613a9-01d4-4d8e-bfa3-3cc8b7e118b3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7q99w" Nov 01 22:33:30.334287 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.334268 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/643613a9-01d4-4d8e-bfa3-3cc8b7e118b3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7q99w\" (UID: \"643613a9-01d4-4d8e-bfa3-3cc8b7e118b3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7q99w" Nov 01 22:33:30.336225 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.336200 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/643613a9-01d4-4d8e-bfa3-3cc8b7e118b3-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7q99w\" (UID: \"643613a9-01d4-4d8e-bfa3-3cc8b7e118b3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7q99w" Nov 01 22:33:30.336446 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.336431 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/643613a9-01d4-4d8e-bfa3-3cc8b7e118b3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7q99w\" (UID: \"643613a9-01d4-4d8e-bfa3-3cc8b7e118b3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7q99w" Nov 01 22:33:30.342115 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.342088 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-499fs\" (UniqueName: \"kubernetes.io/projected/643613a9-01d4-4d8e-bfa3-3cc8b7e118b3-kube-api-access-499fs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7q99w\" (UID: \"643613a9-01d4-4d8e-bfa3-3cc8b7e118b3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7q99w" Nov 01 22:33:30.438503 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.438471 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7q99w" Nov 01 22:33:30.664769 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.664726 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:33:30.719037 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:30.718945 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-7q99w"] Nov 01 22:33:30.722926 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:33:30.722899 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod643613a9_01d4_4d8e_bfa3_3cc8b7e118b3.slice/crio-f646179a02c38d5678aefa6cea07b7989ea4565f8f46c63d4eac118cf64dbaac WatchSource:0}: Error finding container f646179a02c38d5678aefa6cea07b7989ea4565f8f46c63d4eac118cf64dbaac: Status 404 returned error can't find the container with id f646179a02c38d5678aefa6cea07b7989ea4565f8f46c63d4eac118cf64dbaac Nov 01 22:33:31.138978 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:31.138882 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7q99w" event={"ID":"643613a9-01d4-4d8e-bfa3-3cc8b7e118b3","Type":"ContainerStarted","Data":"f646179a02c38d5678aefa6cea07b7989ea4565f8f46c63d4eac118cf64dbaac"} Nov 01 22:33:31.792226 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:31.792186 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:33:32.146429 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:32.146397 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7q99w" event={"ID":"643613a9-01d4-4d8e-bfa3-3cc8b7e118b3","Type":"ContainerStarted","Data":"47e0aaefd624ae85109532c41c929245f135cc2ea1c7e8324a2f4f06aceb06f8"} Nov 01 22:33:32.165637 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:32.165579 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7q99w" podStartSLOduration=0.846084549 podStartE2EDuration="2.165563557s" totalImagesPullingTime="1.319479008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:32.164885419 +0000 UTC m=+580.569476051" watchObservedRunningTime="2026-11-01 22:33:32.165563557 +0000 UTC m=+580.570154188" Nov 01 22:33:32.438956 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:32.438916 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7q99w" Nov 01 22:33:32.517862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:32.517267 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-474rg"] Nov 01 22:33:32.521326 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:32.521298 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-474rg" Nov 01 22:33:32.526046 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:32.525977 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-474rg"] Nov 01 22:33:32.535036 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:32.534619 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 01 22:33:32.654450 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:32.654414 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72f1a530-bd82-48d6-a9a7-020eacd1b1f6-home\") pod \"echo-simulator-kserve-64b6f8f96c-474rg\" (UID: \"72f1a530-bd82-48d6-a9a7-020eacd1b1f6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-474rg" Nov 01 22:33:32.654732 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:32.654702 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f1a530-bd82-48d6-a9a7-020eacd1b1f6-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-474rg\" (UID: \"72f1a530-bd82-48d6-a9a7-020eacd1b1f6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-474rg" Nov 01 22:33:32.654952 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:32.654767 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72f1a530-bd82-48d6-a9a7-020eacd1b1f6-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-474rg\" (UID: \"72f1a530-bd82-48d6-a9a7-020eacd1b1f6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-474rg" Nov 01 22:33:32.654952 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:32.654851 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgcxl\" (UniqueName: \"kubernetes.io/projected/72f1a530-bd82-48d6-a9a7-020eacd1b1f6-kube-api-access-xgcxl\") pod \"echo-simulator-kserve-64b6f8f96c-474rg\" (UID: \"72f1a530-bd82-48d6-a9a7-020eacd1b1f6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-474rg" Nov 01 22:33:32.654952 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:32.654909 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f1a530-bd82-48d6-a9a7-020eacd1b1f6-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-474rg\" (UID: \"72f1a530-bd82-48d6-a9a7-020eacd1b1f6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-474rg" Nov 01 22:33:32.655180 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:32.655049 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f1a530-bd82-48d6-a9a7-020eacd1b1f6-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-474rg\" (UID: \"72f1a530-bd82-48d6-a9a7-020eacd1b1f6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-474rg" Nov 01 22:33:32.756432 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:32.756348 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f1a530-bd82-48d6-a9a7-020eacd1b1f6-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-474rg\" (UID: \"72f1a530-bd82-48d6-a9a7-020eacd1b1f6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-474rg" Nov 01 22:33:32.756432 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:32.756404 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72f1a530-bd82-48d6-a9a7-020eacd1b1f6-home\") pod \"echo-simulator-kserve-64b6f8f96c-474rg\" (UID: \"72f1a530-bd82-48d6-a9a7-020eacd1b1f6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-474rg" Nov 01 22:33:32.756432 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:32.756431 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f1a530-bd82-48d6-a9a7-020eacd1b1f6-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-474rg\" (UID: \"72f1a530-bd82-48d6-a9a7-020eacd1b1f6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-474rg" Nov 01 22:33:32.756659 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:32.756461 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72f1a530-bd82-48d6-a9a7-020eacd1b1f6-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-474rg\" (UID: \"72f1a530-bd82-48d6-a9a7-020eacd1b1f6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-474rg" Nov 01 22:33:32.756659 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:32.756499 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgcxl\" (UniqueName: \"kubernetes.io/projected/72f1a530-bd82-48d6-a9a7-020eacd1b1f6-kube-api-access-xgcxl\") pod \"echo-simulator-kserve-64b6f8f96c-474rg\" (UID: \"72f1a530-bd82-48d6-a9a7-020eacd1b1f6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-474rg" Nov 01 22:33:32.756659 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:32.756541 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f1a530-bd82-48d6-a9a7-020eacd1b1f6-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-474rg\" (UID: \"72f1a530-bd82-48d6-a9a7-020eacd1b1f6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-474rg" Nov 01 22:33:32.756856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:32.756805 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72f1a530-bd82-48d6-a9a7-020eacd1b1f6-home\") pod \"echo-simulator-kserve-64b6f8f96c-474rg\" (UID: \"72f1a530-bd82-48d6-a9a7-020eacd1b1f6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-474rg" Nov 01 22:33:32.756925 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:32.756856 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f1a530-bd82-48d6-a9a7-020eacd1b1f6-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-474rg\" (UID: \"72f1a530-bd82-48d6-a9a7-020eacd1b1f6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-474rg" Nov 01 22:33:32.756979 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:32.756965 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f1a530-bd82-48d6-a9a7-020eacd1b1f6-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-474rg\" (UID: \"72f1a530-bd82-48d6-a9a7-020eacd1b1f6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-474rg" Nov 01 22:33:32.758996 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:32.758974 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f1a530-bd82-48d6-a9a7-020eacd1b1f6-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-474rg\" (UID: \"72f1a530-bd82-48d6-a9a7-020eacd1b1f6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-474rg" Nov 01 22:33:32.759355 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:32.759328 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72f1a530-bd82-48d6-a9a7-020eacd1b1f6-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-474rg\" (UID: \"72f1a530-bd82-48d6-a9a7-020eacd1b1f6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-474rg" Nov 01 22:33:32.771106 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:32.771082 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgcxl\" (UniqueName: \"kubernetes.io/projected/72f1a530-bd82-48d6-a9a7-020eacd1b1f6-kube-api-access-xgcxl\") pod \"echo-simulator-kserve-64b6f8f96c-474rg\" (UID: \"72f1a530-bd82-48d6-a9a7-020eacd1b1f6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-474rg" Nov 01 22:33:32.846146 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:32.846116 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-474rg" Nov 01 22:33:33.118081 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:33.117764 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-474rg"] Nov 01 22:33:33.120722 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:33:33.120689 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f1a530_bd82_48d6_a9a7_020eacd1b1f6.slice/crio-acb56e703fbbc94f50160f3419fc0e3083a75af4d253c0734a9c9f010d118652 WatchSource:0}: Error finding container acb56e703fbbc94f50160f3419fc0e3083a75af4d253c0734a9c9f010d118652: Status 404 returned error can't find the container with id acb56e703fbbc94f50160f3419fc0e3083a75af4d253c0734a9c9f010d118652 Nov 01 22:33:33.152664 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:33.152624 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-474rg" event={"ID":"72f1a530-bd82-48d6-a9a7-020eacd1b1f6","Type":"ContainerStarted","Data":"acb56e703fbbc94f50160f3419fc0e3083a75af4d253c0734a9c9f010d118652"} Nov 01 22:33:33.622627 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:33.622591 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr"] Nov 01 22:33:33.626120 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:33.626094 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr" Nov 01 22:33:33.632016 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:33.631977 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 01 22:33:33.639326 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:33.639267 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr"] Nov 01 22:33:33.767993 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:33.767957 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c42d6ad-2b35-4217-b81e-18e3b79be22e-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr\" (UID: \"5c42d6ad-2b35-4217-b81e-18e3b79be22e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr" Nov 01 22:33:33.768171 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:33.768002 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbclq\" (UniqueName: \"kubernetes.io/projected/5c42d6ad-2b35-4217-b81e-18e3b79be22e-kube-api-access-wbclq\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr\" (UID: \"5c42d6ad-2b35-4217-b81e-18e3b79be22e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr" Nov 01 22:33:33.768171 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:33.768100 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c42d6ad-2b35-4217-b81e-18e3b79be22e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr\" (UID: \"5c42d6ad-2b35-4217-b81e-18e3b79be22e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr" Nov 01 22:33:33.768256 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:33.768185 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c42d6ad-2b35-4217-b81e-18e3b79be22e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr\" (UID: \"5c42d6ad-2b35-4217-b81e-18e3b79be22e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr" Nov 01 22:33:33.768256 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:33.768243 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c42d6ad-2b35-4217-b81e-18e3b79be22e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr\" (UID: \"5c42d6ad-2b35-4217-b81e-18e3b79be22e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr" Nov 01 22:33:33.768327 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:33.768262 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c42d6ad-2b35-4217-b81e-18e3b79be22e-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr\" (UID: \"5c42d6ad-2b35-4217-b81e-18e3b79be22e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr" Nov 01 22:33:33.869864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:33.869801 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c42d6ad-2b35-4217-b81e-18e3b79be22e-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr\" (UID: \"5c42d6ad-2b35-4217-b81e-18e3b79be22e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr" Nov 01 22:33:33.870018 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:33.869880 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbclq\" (UniqueName: \"kubernetes.io/projected/5c42d6ad-2b35-4217-b81e-18e3b79be22e-kube-api-access-wbclq\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr\" (UID: \"5c42d6ad-2b35-4217-b81e-18e3b79be22e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr" Nov 01 22:33:33.870018 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:33.869915 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c42d6ad-2b35-4217-b81e-18e3b79be22e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr\" (UID: \"5c42d6ad-2b35-4217-b81e-18e3b79be22e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr" Nov 01 22:33:33.870018 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:33.869958 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c42d6ad-2b35-4217-b81e-18e3b79be22e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr\" (UID: \"5c42d6ad-2b35-4217-b81e-18e3b79be22e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr" Nov 01 22:33:33.870167 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:33.870033 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c42d6ad-2b35-4217-b81e-18e3b79be22e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr\" (UID: \"5c42d6ad-2b35-4217-b81e-18e3b79be22e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr" Nov 01 22:33:33.870167 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:33.870061 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c42d6ad-2b35-4217-b81e-18e3b79be22e-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr\" (UID: \"5c42d6ad-2b35-4217-b81e-18e3b79be22e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr" Nov 01 22:33:33.870296 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:33.870257 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c42d6ad-2b35-4217-b81e-18e3b79be22e-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr\" (UID: \"5c42d6ad-2b35-4217-b81e-18e3b79be22e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr" Nov 01 22:33:33.870333 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:33.870291 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c42d6ad-2b35-4217-b81e-18e3b79be22e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr\" (UID: \"5c42d6ad-2b35-4217-b81e-18e3b79be22e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr" Nov 01 22:33:33.870601 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:33.870564 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c42d6ad-2b35-4217-b81e-18e3b79be22e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr\" (UID: \"5c42d6ad-2b35-4217-b81e-18e3b79be22e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr" Nov 01 22:33:33.872715 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:33.872657 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c42d6ad-2b35-4217-b81e-18e3b79be22e-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr\" (UID: \"5c42d6ad-2b35-4217-b81e-18e3b79be22e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr" Nov 01 22:33:33.872715 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:33.872681 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c42d6ad-2b35-4217-b81e-18e3b79be22e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr\" (UID: \"5c42d6ad-2b35-4217-b81e-18e3b79be22e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr" Nov 01 22:33:33.879071 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:33.879043 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbclq\" (UniqueName: \"kubernetes.io/projected/5c42d6ad-2b35-4217-b81e-18e3b79be22e-kube-api-access-wbclq\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr\" (UID: \"5c42d6ad-2b35-4217-b81e-18e3b79be22e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr" Nov 01 22:33:33.955015 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:33.954976 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr" Nov 01 22:33:34.246144 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:34.245164 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr"] Nov 01 22:33:34.252450 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:33:34.252408 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c42d6ad_2b35_4217_b81e_18e3b79be22e.slice/crio-14f35b2b2f05e978f65b056e3fbd73ec2ed88ea7676f4e7e3b20529a8000c2d8 WatchSource:0}: Error finding container 14f35b2b2f05e978f65b056e3fbd73ec2ed88ea7676f4e7e3b20529a8000c2d8: Status 404 returned error can't find the container with id 14f35b2b2f05e978f65b056e3fbd73ec2ed88ea7676f4e7e3b20529a8000c2d8 Nov 01 22:33:34.487595 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:34.487557 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:33:35.165026 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:35.164940 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-474rg" event={"ID":"72f1a530-bd82-48d6-a9a7-020eacd1b1f6","Type":"ContainerStarted","Data":"0f5f60fc212d8baad40fd56328e88261aa776003f3ec27dc1a2dba8f14af673a"} Nov 01 22:33:35.166514 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:35.166486 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr" event={"ID":"5c42d6ad-2b35-4217-b81e-18e3b79be22e","Type":"ContainerStarted","Data":"88d893698555b20e378ba52b3871354d911beba04014fbfa272f9cef2bded392"} Nov 01 22:33:35.166631 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:35.166539 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr" event={"ID":"5c42d6ad-2b35-4217-b81e-18e3b79be22e","Type":"ContainerStarted","Data":"14f35b2b2f05e978f65b056e3fbd73ec2ed88ea7676f4e7e3b20529a8000c2d8"} Nov 01 22:33:35.183536 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:35.183482 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-474rg" podStartSLOduration=1.447824964 podStartE2EDuration="3.18346474s" totalImagesPullingTime="1.735639776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:35.182778537 +0000 UTC m=+583.587369173" watchObservedRunningTime="2026-11-01 22:33:35.18346474 +0000 UTC m=+583.588055376" Nov 01 22:33:35.199639 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:35.199584 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr" podStartSLOduration=1.9168902490000002 podStartE2EDuration="2.199571158s" totalImagesPullingTime="282.680909ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:35.198737267 +0000 UTC m=+583.603327905" watchObservedRunningTime="2026-11-01 22:33:35.199571158 +0000 UTC m=+583.604161814" Nov 01 22:33:35.846654 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:35.846617 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-474rg" Nov 01 22:33:35.955703 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:35.955670 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr" Nov 01 22:33:36.478151 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:36.478116 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-67p7h"] Nov 01 22:33:36.483400 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:36.482241 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67p7h" Nov 01 22:33:36.487541 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:36.487512 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 01 22:33:36.488821 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:36.488795 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-67p7h"] Nov 01 22:33:36.601721 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:36.601686 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s92fn\" (UniqueName: \"kubernetes.io/projected/cf198325-a260-434e-a504-f73add235e9b-kube-api-access-s92fn\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67p7h\" (UID: \"cf198325-a260-434e-a504-f73add235e9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67p7h" Nov 01 22:33:36.601939 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:36.601735 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf198325-a260-434e-a504-f73add235e9b-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67p7h\" (UID: \"cf198325-a260-434e-a504-f73add235e9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67p7h" Nov 01 22:33:36.601939 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:36.601862 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf198325-a260-434e-a504-f73add235e9b-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67p7h\" (UID: \"cf198325-a260-434e-a504-f73add235e9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67p7h" Nov 01 22:33:36.602043 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:36.601933 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf198325-a260-434e-a504-f73add235e9b-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67p7h\" (UID: \"cf198325-a260-434e-a504-f73add235e9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67p7h" Nov 01 22:33:36.602043 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:36.602033 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf198325-a260-434e-a504-f73add235e9b-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67p7h\" (UID: \"cf198325-a260-434e-a504-f73add235e9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67p7h" Nov 01 22:33:36.602190 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:36.602174 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf198325-a260-434e-a504-f73add235e9b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67p7h\" (UID: \"cf198325-a260-434e-a504-f73add235e9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67p7h" Nov 01 22:33:36.703403 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:36.703361 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf198325-a260-434e-a504-f73add235e9b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67p7h\" (UID: \"cf198325-a260-434e-a504-f73add235e9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67p7h" Nov 01 22:33:36.703600 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:36.703439 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s92fn\" (UniqueName: \"kubernetes.io/projected/cf198325-a260-434e-a504-f73add235e9b-kube-api-access-s92fn\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67p7h\" (UID: \"cf198325-a260-434e-a504-f73add235e9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67p7h" Nov 01 22:33:36.703600 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:36.703463 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf198325-a260-434e-a504-f73add235e9b-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67p7h\" (UID: \"cf198325-a260-434e-a504-f73add235e9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67p7h" Nov 01 22:33:36.703600 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:36.703487 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf198325-a260-434e-a504-f73add235e9b-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67p7h\" (UID: \"cf198325-a260-434e-a504-f73add235e9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67p7h" Nov 01 22:33:36.703600 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:36.703502 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf198325-a260-434e-a504-f73add235e9b-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67p7h\" (UID: \"cf198325-a260-434e-a504-f73add235e9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67p7h" Nov 01 22:33:36.703600 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:36.703528 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf198325-a260-434e-a504-f73add235e9b-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67p7h\" (UID: \"cf198325-a260-434e-a504-f73add235e9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67p7h" Nov 01 22:33:36.703885 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:36.703819 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf198325-a260-434e-a504-f73add235e9b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67p7h\" (UID: \"cf198325-a260-434e-a504-f73add235e9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67p7h" Nov 01 22:33:36.703930 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:36.703860 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf198325-a260-434e-a504-f73add235e9b-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67p7h\" (UID: \"cf198325-a260-434e-a504-f73add235e9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67p7h" Nov 01 22:33:36.703970 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:36.703928 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf198325-a260-434e-a504-f73add235e9b-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67p7h\" (UID: \"cf198325-a260-434e-a504-f73add235e9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67p7h" Nov 01 22:33:36.706075 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:36.706052 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf198325-a260-434e-a504-f73add235e9b-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67p7h\" (UID: \"cf198325-a260-434e-a504-f73add235e9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67p7h" Nov 01 22:33:36.706357 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:36.706334 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf198325-a260-434e-a504-f73add235e9b-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67p7h\" (UID: \"cf198325-a260-434e-a504-f73add235e9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67p7h" Nov 01 22:33:36.720787 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:36.720754 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s92fn\" (UniqueName: \"kubernetes.io/projected/cf198325-a260-434e-a504-f73add235e9b-kube-api-access-s92fn\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-67p7h\" (UID: \"cf198325-a260-434e-a504-f73add235e9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67p7h" Nov 01 22:33:36.799774 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:36.799690 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67p7h" Nov 01 22:33:37.074817 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:37.074746 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-67p7h"] Nov 01 22:33:37.077697 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:33:37.077654 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf198325_a260_434e_a504_f73add235e9b.slice/crio-23805d34ecca132bfce99709890b1372132709bfaa517899d98f8fc940dc1488 WatchSource:0}: Error finding container 23805d34ecca132bfce99709890b1372132709bfaa517899d98f8fc940dc1488: Status 404 returned error can't find the container with id 23805d34ecca132bfce99709890b1372132709bfaa517899d98f8fc940dc1488 Nov 01 22:33:37.177101 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:37.177065 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67p7h" event={"ID":"cf198325-a260-434e-a504-f73add235e9b","Type":"ContainerStarted","Data":"23805d34ecca132bfce99709890b1372132709bfaa517899d98f8fc940dc1488"} Nov 01 22:33:38.183092 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:38.183055 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67p7h" event={"ID":"cf198325-a260-434e-a504-f73add235e9b","Type":"ContainerStarted","Data":"c3a0e0644f594b3327609fb6dd47341697cbd46bfadfa51eaeedc69fa4397f27"} Nov 01 22:33:38.201559 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:38.201511 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67p7h" podStartSLOduration=1.917442831 podStartE2EDuration="2.201498243s" totalImagesPullingTime="284.055412ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:38.200647965 +0000 UTC m=+586.605238598" watchObservedRunningTime="2026-11-01 22:33:38.201498243 +0000 UTC m=+586.606088889" Nov 01 22:33:38.307773 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:38.307736 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:33:38.799985 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:38.799956 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67p7h" Nov 01 22:33:39.243562 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:39.243527 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-rpcvn"] Nov 01 22:33:39.247751 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:39.247722 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpcvn" Nov 01 22:33:39.259460 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:39.259436 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 01 22:33:39.267449 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:39.267417 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-rpcvn"] Nov 01 22:33:39.429406 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:39.429373 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b7cef02-a6d1-41ba-9bfd-9fe36d40f249-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpcvn\" (UID: \"3b7cef02-a6d1-41ba-9bfd-9fe36d40f249\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpcvn" Nov 01 22:33:39.429591 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:39.429417 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b7cef02-a6d1-41ba-9bfd-9fe36d40f249-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpcvn\" (UID: \"3b7cef02-a6d1-41ba-9bfd-9fe36d40f249\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpcvn" Nov 01 22:33:39.429591 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:39.429434 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b7cef02-a6d1-41ba-9bfd-9fe36d40f249-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpcvn\" (UID: \"3b7cef02-a6d1-41ba-9bfd-9fe36d40f249\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpcvn" Nov 01 22:33:39.429591 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:39.429456 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b7cef02-a6d1-41ba-9bfd-9fe36d40f249-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpcvn\" (UID: \"3b7cef02-a6d1-41ba-9bfd-9fe36d40f249\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpcvn" Nov 01 22:33:39.429697 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:39.429606 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b7cef02-a6d1-41ba-9bfd-9fe36d40f249-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpcvn\" (UID: \"3b7cef02-a6d1-41ba-9bfd-9fe36d40f249\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpcvn" Nov 01 22:33:39.429697 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:39.429639 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8zzd\" (UniqueName: \"kubernetes.io/projected/3b7cef02-a6d1-41ba-9bfd-9fe36d40f249-kube-api-access-r8zzd\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpcvn\" (UID: \"3b7cef02-a6d1-41ba-9bfd-9fe36d40f249\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpcvn" Nov 01 22:33:39.530947 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:39.530825 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b7cef02-a6d1-41ba-9bfd-9fe36d40f249-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpcvn\" (UID: \"3b7cef02-a6d1-41ba-9bfd-9fe36d40f249\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpcvn" Nov 01 22:33:39.530947 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:39.530927 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b7cef02-a6d1-41ba-9bfd-9fe36d40f249-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpcvn\" (UID: \"3b7cef02-a6d1-41ba-9bfd-9fe36d40f249\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpcvn" Nov 01 22:33:39.531171 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:39.530950 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b7cef02-a6d1-41ba-9bfd-9fe36d40f249-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpcvn\" (UID: \"3b7cef02-a6d1-41ba-9bfd-9fe36d40f249\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpcvn" Nov 01 22:33:39.531171 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:39.530974 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b7cef02-a6d1-41ba-9bfd-9fe36d40f249-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpcvn\" (UID: \"3b7cef02-a6d1-41ba-9bfd-9fe36d40f249\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpcvn" Nov 01 22:33:39.531171 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:39.530995 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b7cef02-a6d1-41ba-9bfd-9fe36d40f249-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpcvn\" (UID: \"3b7cef02-a6d1-41ba-9bfd-9fe36d40f249\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpcvn" Nov 01 22:33:39.531171 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:39.531119 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8zzd\" (UniqueName: \"kubernetes.io/projected/3b7cef02-a6d1-41ba-9bfd-9fe36d40f249-kube-api-access-r8zzd\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpcvn\" (UID: \"3b7cef02-a6d1-41ba-9bfd-9fe36d40f249\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpcvn" Nov 01 22:33:39.531365 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:39.531315 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b7cef02-a6d1-41ba-9bfd-9fe36d40f249-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpcvn\" (UID: \"3b7cef02-a6d1-41ba-9bfd-9fe36d40f249\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpcvn" Nov 01 22:33:39.531482 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:39.531454 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b7cef02-a6d1-41ba-9bfd-9fe36d40f249-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpcvn\" (UID: \"3b7cef02-a6d1-41ba-9bfd-9fe36d40f249\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpcvn" Nov 01 22:33:39.531532 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:39.531480 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b7cef02-a6d1-41ba-9bfd-9fe36d40f249-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpcvn\" (UID: \"3b7cef02-a6d1-41ba-9bfd-9fe36d40f249\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpcvn" Nov 01 22:33:39.533578 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:39.533542 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b7cef02-a6d1-41ba-9bfd-9fe36d40f249-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpcvn\" (UID: \"3b7cef02-a6d1-41ba-9bfd-9fe36d40f249\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpcvn" Nov 01 22:33:39.533801 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:39.533785 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b7cef02-a6d1-41ba-9bfd-9fe36d40f249-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpcvn\" (UID: \"3b7cef02-a6d1-41ba-9bfd-9fe36d40f249\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpcvn" Nov 01 22:33:39.539778 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:39.539755 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8zzd\" (UniqueName: \"kubernetes.io/projected/3b7cef02-a6d1-41ba-9bfd-9fe36d40f249-kube-api-access-r8zzd\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rpcvn\" (UID: \"3b7cef02-a6d1-41ba-9bfd-9fe36d40f249\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpcvn" Nov 01 22:33:39.567587 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:39.567552 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpcvn" Nov 01 22:33:39.847142 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:39.846978 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-rpcvn"] Nov 01 22:33:39.849959 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:33:39.849925 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7cef02_a6d1_41ba_9bfd_9fe36d40f249.slice/crio-436f29329db431123336954a200ffc77a628ff77b2799aaa5f733d521d2eb306 WatchSource:0}: Error finding container 436f29329db431123336954a200ffc77a628ff77b2799aaa5f733d521d2eb306: Status 404 returned error can't find the container with id 436f29329db431123336954a200ffc77a628ff77b2799aaa5f733d521d2eb306 Nov 01 22:33:40.193130 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:40.193103 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpcvn" event={"ID":"3b7cef02-a6d1-41ba-9bfd-9fe36d40f249","Type":"ContainerStarted","Data":"436f29329db431123336954a200ffc77a628ff77b2799aaa5f733d521d2eb306"} Nov 01 22:33:40.438820 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:40.438734 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7q99w" Nov 01 22:33:40.451390 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:40.451359 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7q99w" Nov 01 22:33:41.201061 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:41.201017 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpcvn" event={"ID":"3b7cef02-a6d1-41ba-9bfd-9fe36d40f249","Type":"ContainerStarted","Data":"8dd66b20a9071eddd2c9781c59b43eba5b94841ca21bd4a1a77aeba97b8c6079"} Nov 01 22:33:41.213419 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:41.213388 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7q99w" Nov 01 22:33:41.222420 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:41.222365 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpcvn" podStartSLOduration=1.884499801 podStartE2EDuration="2.222345471s" totalImagesPullingTime="337.84567ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:41.220931305 +0000 UTC m=+589.625521939" watchObservedRunningTime="2026-11-01 22:33:41.222345471 +0000 UTC m=+589.626936103" Nov 01 22:33:41.568630 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:41.568543 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpcvn" Nov 01 22:33:42.847362 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:42.847308 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-474rg" Nov 01 22:33:42.859885 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:42.859853 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-474rg" Nov 01 22:33:43.220741 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:43.220711 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-474rg" Nov 01 22:33:43.521154 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:43.521057 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:33:43.955773 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:43.955742 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr" Nov 01 22:33:43.968031 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:43.968004 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr" Nov 01 22:33:44.224752 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:44.224674 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr" Nov 01 22:33:46.800687 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:46.800653 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67p7h" Nov 01 22:33:46.813281 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:46.813252 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67p7h" Nov 01 22:33:46.959667 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:33:46.959634 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:33:47.237148 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:47.237123 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67p7h" Nov 01 22:33:49.567918 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:49.567883 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpcvn" Nov 01 22:33:49.580603 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:49.580582 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpcvn" Nov 01 22:33:49.648607 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:49.648575 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-68ccc5d4b4-cs5zt"] Nov 01 22:33:49.652383 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:49.652361 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68ccc5d4b4-cs5zt" Nov 01 22:33:49.659106 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:49.659080 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68ccc5d4b4-cs5zt"] Nov 01 22:33:49.718606 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:49.718563 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ac80b35e-3170-4fba-9039-b51ba2f236ec-tls-cert\") pod \"authorino-68ccc5d4b4-cs5zt\" (UID: \"ac80b35e-3170-4fba-9039-b51ba2f236ec\") " pod="kuadrant-system/authorino-68ccc5d4b4-cs5zt" Nov 01 22:33:49.718796 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:49.718661 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7ddl\" (UniqueName: \"kubernetes.io/projected/ac80b35e-3170-4fba-9039-b51ba2f236ec-kube-api-access-n7ddl\") pod \"authorino-68ccc5d4b4-cs5zt\" (UID: \"ac80b35e-3170-4fba-9039-b51ba2f236ec\") " pod="kuadrant-system/authorino-68ccc5d4b4-cs5zt" Nov 01 22:33:49.819238 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:49.819144 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ac80b35e-3170-4fba-9039-b51ba2f236ec-tls-cert\") pod \"authorino-68ccc5d4b4-cs5zt\" (UID: \"ac80b35e-3170-4fba-9039-b51ba2f236ec\") " pod="kuadrant-system/authorino-68ccc5d4b4-cs5zt" Nov 01 22:33:49.819238 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:49.819200 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7ddl\" (UniqueName: \"kubernetes.io/projected/ac80b35e-3170-4fba-9039-b51ba2f236ec-kube-api-access-n7ddl\") pod \"authorino-68ccc5d4b4-cs5zt\" (UID: \"ac80b35e-3170-4fba-9039-b51ba2f236ec\") " pod="kuadrant-system/authorino-68ccc5d4b4-cs5zt" Nov 01 22:33:49.821752 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:49.821726 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ac80b35e-3170-4fba-9039-b51ba2f236ec-tls-cert\") pod \"authorino-68ccc5d4b4-cs5zt\" (UID: \"ac80b35e-3170-4fba-9039-b51ba2f236ec\") " pod="kuadrant-system/authorino-68ccc5d4b4-cs5zt" Nov 01 22:33:49.827394 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:49.827372 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7ddl\" (UniqueName: \"kubernetes.io/projected/ac80b35e-3170-4fba-9039-b51ba2f236ec-kube-api-access-n7ddl\") pod \"authorino-68ccc5d4b4-cs5zt\" (UID: \"ac80b35e-3170-4fba-9039-b51ba2f236ec\") " pod="kuadrant-system/authorino-68ccc5d4b4-cs5zt" Nov 01 22:33:49.965855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:49.965796 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68ccc5d4b4-cs5zt" Nov 01 22:33:50.238025 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:50.238002 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68ccc5d4b4-cs5zt"] Nov 01 22:33:50.240785 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:33:50.240757 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac80b35e_3170_4fba_9039_b51ba2f236ec.slice/crio-ebc799e2dda9f5ab01fe90d0e27c993adaf92d0d26ca34fb0cfd9dc1643aed2a WatchSource:0}: Error finding container ebc799e2dda9f5ab01fe90d0e27c993adaf92d0d26ca34fb0cfd9dc1643aed2a: Status 404 returned error can't find the container with id ebc799e2dda9f5ab01fe90d0e27c993adaf92d0d26ca34fb0cfd9dc1643aed2a Nov 01 22:33:50.251972 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:50.251951 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rpcvn" Nov 01 22:33:51.245809 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:51.245773 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68ccc5d4b4-cs5zt" event={"ID":"ac80b35e-3170-4fba-9039-b51ba2f236ec","Type":"ContainerStarted","Data":"ebc799e2dda9f5ab01fe90d0e27c993adaf92d0d26ca34fb0cfd9dc1643aed2a"} Nov 01 22:33:52.251932 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:52.251898 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68ccc5d4b4-cs5zt" event={"ID":"ac80b35e-3170-4fba-9039-b51ba2f236ec","Type":"ContainerStarted","Data":"dc43d0b098b33ec58097a8556cd4e32fd46e7b26544d5a450a23a02401c5cf6d"} Nov 01 22:33:52.284804 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:52.284740 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-68ccc5d4b4-cs5zt" podStartSLOduration=2.052749561 podStartE2EDuration="3.28472036s" totalImagesPullingTime="1.231970799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:52.283963223 +0000 UTC m=+600.688553877" watchObservedRunningTime="2026-11-01 22:33:52.28472036 +0000 UTC m=+600.689310993" Nov 01 22:33:52.360731 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:52.358429 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bb7596679-xvxvn"] Nov 01 22:33:52.360731 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:52.358673 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7bb7596679-xvxvn" podUID="d23bde01-3662-4d8c-8581-afcd8d7f74e1" containerName="authorino" containerID="cri-o://5edb8b05546ee4bfd22cb2c021f2dab243ff66446f8761b60965a9e0fd6cfc2c" gracePeriod=30 Nov 01 22:33:52.637034 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:52.637008 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bb7596679-xvxvn" Nov 01 22:33:52.646231 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:52.646202 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d23bde01-3662-4d8c-8581-afcd8d7f74e1-tls-cert\" (UniqueName: \"kubernetes.io/secret/d23bde01-3662-4d8c-8581-afcd8d7f74e1-tls-cert\") pod \"d23bde01-3662-4d8c-8581-afcd8d7f74e1\" (UID: \"d23bde01-3662-4d8c-8581-afcd8d7f74e1\") " Nov 01 22:33:52.646392 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:52.646250 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d23bde01-3662-4d8c-8581-afcd8d7f74e1-kube-api-access-dzwsm\" (UniqueName: \"kubernetes.io/projected/d23bde01-3662-4d8c-8581-afcd8d7f74e1-kube-api-access-dzwsm\") pod \"d23bde01-3662-4d8c-8581-afcd8d7f74e1\" (UID: \"d23bde01-3662-4d8c-8581-afcd8d7f74e1\") " Nov 01 22:33:52.648903 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:52.648876 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d23bde01-3662-4d8c-8581-afcd8d7f74e1-kube-api-access-dzwsm" pod "d23bde01-3662-4d8c-8581-afcd8d7f74e1" (UID: "d23bde01-3662-4d8c-8581-afcd8d7f74e1"). InnerVolumeSpecName "kube-api-access-dzwsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:33:52.660745 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:52.660674 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d23bde01-3662-4d8c-8581-afcd8d7f74e1-tls-cert" pod "d23bde01-3662-4d8c-8581-afcd8d7f74e1" (UID: "d23bde01-3662-4d8c-8581-afcd8d7f74e1"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:33:52.747430 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:52.747392 5011 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d23bde01-3662-4d8c-8581-afcd8d7f74e1-tls-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:52.747430 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:52.747421 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzwsm\" (UniqueName: \"kubernetes.io/projected/d23bde01-3662-4d8c-8581-afcd8d7f74e1-kube-api-access-dzwsm\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:33:53.258262 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:53.258225 5011 generic.go:356] "Generic (PLEG): container finished" podID="d23bde01-3662-4d8c-8581-afcd8d7f74e1" containerID="5edb8b05546ee4bfd22cb2c021f2dab243ff66446f8761b60965a9e0fd6cfc2c" exitCode=0 Nov 01 22:33:53.258707 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:53.258267 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bb7596679-xvxvn" Nov 01 22:33:53.258707 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:53.258306 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bb7596679-xvxvn" event={"ID":"d23bde01-3662-4d8c-8581-afcd8d7f74e1","Type":"ContainerDied","Data":"5edb8b05546ee4bfd22cb2c021f2dab243ff66446f8761b60965a9e0fd6cfc2c"} Nov 01 22:33:53.258707 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:53.258346 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bb7596679-xvxvn" event={"ID":"d23bde01-3662-4d8c-8581-afcd8d7f74e1","Type":"ContainerDied","Data":"a5a53a05859b7d9189bc8d69d8d5c78170aaf6b19d7be0a1ea9e1605bd8c7a32"} Nov 01 22:33:53.258707 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:53.258376 5011 scope.go:122] "RemoveContainer" containerID="5edb8b05546ee4bfd22cb2c021f2dab243ff66446f8761b60965a9e0fd6cfc2c" Nov 01 22:33:53.268293 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:53.268275 5011 scope.go:122] "RemoveContainer" containerID="5edb8b05546ee4bfd22cb2c021f2dab243ff66446f8761b60965a9e0fd6cfc2c" Nov 01 22:33:53.268596 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:33:53.268573 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5edb8b05546ee4bfd22cb2c021f2dab243ff66446f8761b60965a9e0fd6cfc2c\": container with ID starting with 5edb8b05546ee4bfd22cb2c021f2dab243ff66446f8761b60965a9e0fd6cfc2c not found: ID does not exist" containerID="5edb8b05546ee4bfd22cb2c021f2dab243ff66446f8761b60965a9e0fd6cfc2c" Nov 01 22:33:53.268666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:53.268603 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5edb8b05546ee4bfd22cb2c021f2dab243ff66446f8761b60965a9e0fd6cfc2c"} err="failed to get container status \"5edb8b05546ee4bfd22cb2c021f2dab243ff66446f8761b60965a9e0fd6cfc2c\": rpc error: code = NotFound desc = could not find container \"5edb8b05546ee4bfd22cb2c021f2dab243ff66446f8761b60965a9e0fd6cfc2c\": container with ID starting with 5edb8b05546ee4bfd22cb2c021f2dab243ff66446f8761b60965a9e0fd6cfc2c not found: ID does not exist" Nov 01 22:33:53.280395 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:53.280366 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bb7596679-xvxvn"] Nov 01 22:33:53.284065 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:53.284042 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bb7596679-xvxvn"] Nov 01 22:33:53.745176 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:33:53.745147 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d23bde01-3662-4d8c-8581-afcd8d7f74e1" path="/var/lib/kubelet/pods/d23bde01-3662-4d8c-8581-afcd8d7f74e1/volumes" Nov 01 22:34:01.020259 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.020225 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s"] Nov 01 22:34:01.020653 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.020604 5011 cpu_manager.go:414] "containerMap: removing container" podUID="d23bde01-3662-4d8c-8581-afcd8d7f74e1" containerName="authorino" Nov 01 22:34:01.020653 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.020619 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d23bde01-3662-4d8c-8581-afcd8d7f74e1" containerName="authorino" Nov 01 22:34:01.020735 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.020696 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="d23bde01-3662-4d8c-8581-afcd8d7f74e1" containerName="authorino" Nov 01 22:34:01.025310 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.025281 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:01.045263 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.045215 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s"] Nov 01 22:34:01.120633 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.120594 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-27s9n\"" Nov 01 22:34:01.132173 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.132140 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/734444e5-d903-47e2-9ed0-d7617fd3b2ed-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9x5s\" (UID: \"734444e5-d903-47e2-9ed0-d7617fd3b2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:01.132173 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.132174 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/734444e5-d903-47e2-9ed0-d7617fd3b2ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9x5s\" (UID: \"734444e5-d903-47e2-9ed0-d7617fd3b2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:01.132408 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.132266 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/734444e5-d903-47e2-9ed0-d7617fd3b2ed-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9x5s\" (UID: \"734444e5-d903-47e2-9ed0-d7617fd3b2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:01.132408 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.132305 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/734444e5-d903-47e2-9ed0-d7617fd3b2ed-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9x5s\" (UID: \"734444e5-d903-47e2-9ed0-d7617fd3b2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:01.132408 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.132343 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/734444e5-d903-47e2-9ed0-d7617fd3b2ed-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9x5s\" (UID: \"734444e5-d903-47e2-9ed0-d7617fd3b2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:01.132408 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.132376 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/734444e5-d903-47e2-9ed0-d7617fd3b2ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9x5s\" (UID: \"734444e5-d903-47e2-9ed0-d7617fd3b2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:01.132529 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.132467 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/734444e5-d903-47e2-9ed0-d7617fd3b2ed-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9x5s\" (UID: \"734444e5-d903-47e2-9ed0-d7617fd3b2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:01.132529 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.132491 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j475t\" (UniqueName: \"kubernetes.io/projected/734444e5-d903-47e2-9ed0-d7617fd3b2ed-kube-api-access-j475t\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9x5s\" (UID: \"734444e5-d903-47e2-9ed0-d7617fd3b2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:01.132590 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.132559 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/734444e5-d903-47e2-9ed0-d7617fd3b2ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9x5s\" (UID: \"734444e5-d903-47e2-9ed0-d7617fd3b2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:01.233514 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.233473 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/734444e5-d903-47e2-9ed0-d7617fd3b2ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9x5s\" (UID: \"734444e5-d903-47e2-9ed0-d7617fd3b2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:01.233739 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.233537 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/734444e5-d903-47e2-9ed0-d7617fd3b2ed-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9x5s\" (UID: \"734444e5-d903-47e2-9ed0-d7617fd3b2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:01.233739 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.233553 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/734444e5-d903-47e2-9ed0-d7617fd3b2ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9x5s\" (UID: \"734444e5-d903-47e2-9ed0-d7617fd3b2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:01.233739 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.233596 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/734444e5-d903-47e2-9ed0-d7617fd3b2ed-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9x5s\" (UID: \"734444e5-d903-47e2-9ed0-d7617fd3b2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:01.233739 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.233622 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/734444e5-d903-47e2-9ed0-d7617fd3b2ed-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9x5s\" (UID: \"734444e5-d903-47e2-9ed0-d7617fd3b2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:01.233739 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.233637 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/734444e5-d903-47e2-9ed0-d7617fd3b2ed-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9x5s\" (UID: \"734444e5-d903-47e2-9ed0-d7617fd3b2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:01.233739 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.233657 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/734444e5-d903-47e2-9ed0-d7617fd3b2ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9x5s\" (UID: \"734444e5-d903-47e2-9ed0-d7617fd3b2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:01.233739 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.233719 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/734444e5-d903-47e2-9ed0-d7617fd3b2ed-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9x5s\" (UID: \"734444e5-d903-47e2-9ed0-d7617fd3b2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:01.234133 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.233752 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j475t\" (UniqueName: \"kubernetes.io/projected/734444e5-d903-47e2-9ed0-d7617fd3b2ed-kube-api-access-j475t\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9x5s\" (UID: \"734444e5-d903-47e2-9ed0-d7617fd3b2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:01.234133 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.233991 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/734444e5-d903-47e2-9ed0-d7617fd3b2ed-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9x5s\" (UID: \"734444e5-d903-47e2-9ed0-d7617fd3b2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:01.234133 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.234015 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/734444e5-d903-47e2-9ed0-d7617fd3b2ed-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9x5s\" (UID: \"734444e5-d903-47e2-9ed0-d7617fd3b2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:01.234133 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.234074 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/734444e5-d903-47e2-9ed0-d7617fd3b2ed-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9x5s\" (UID: \"734444e5-d903-47e2-9ed0-d7617fd3b2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:01.234345 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.234289 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/734444e5-d903-47e2-9ed0-d7617fd3b2ed-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9x5s\" (UID: \"734444e5-d903-47e2-9ed0-d7617fd3b2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:01.234597 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.234566 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/734444e5-d903-47e2-9ed0-d7617fd3b2ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9x5s\" (UID: \"734444e5-d903-47e2-9ed0-d7617fd3b2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:01.236150 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.236129 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/734444e5-d903-47e2-9ed0-d7617fd3b2ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9x5s\" (UID: \"734444e5-d903-47e2-9ed0-d7617fd3b2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:01.236448 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.236427 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/734444e5-d903-47e2-9ed0-d7617fd3b2ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9x5s\" (UID: \"734444e5-d903-47e2-9ed0-d7617fd3b2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:01.242356 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.242333 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/734444e5-d903-47e2-9ed0-d7617fd3b2ed-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9x5s\" (UID: \"734444e5-d903-47e2-9ed0-d7617fd3b2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:01.248012 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.247990 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j475t\" (UniqueName: \"kubernetes.io/projected/734444e5-d903-47e2-9ed0-d7617fd3b2ed-kube-api-access-j475t\") pod \"maas-default-gateway-openshift-default-6886b96f9-z9x5s\" (UID: \"734444e5-d903-47e2-9ed0-d7617fd3b2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:01.338740 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.338664 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:01.619017 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:01.618994 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s"] Nov 01 22:34:01.621391 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:34:01.621365 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod734444e5_d903_47e2_9ed0_d7617fd3b2ed.slice/crio-a463af0b4677ea72546c2c431ede59fdcbc0b35bfca953cc541be25987ea65a5 WatchSource:0}: Error finding container a463af0b4677ea72546c2c431ede59fdcbc0b35bfca953cc541be25987ea65a5: Status 404 returned error can't find the container with id a463af0b4677ea72546c2c431ede59fdcbc0b35bfca953cc541be25987ea65a5 Nov 01 22:34:02.301552 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:02.301515 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" event={"ID":"734444e5-d903-47e2-9ed0-d7617fd3b2ed","Type":"ContainerStarted","Data":"a463af0b4677ea72546c2c431ede59fdcbc0b35bfca953cc541be25987ea65a5"} Nov 01 22:34:05.402399 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:05.402369 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:34:05.402650 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:05.402437 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:34:05.402650 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:05.402465 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:34:06.321992 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:06.321957 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" event={"ID":"734444e5-d903-47e2-9ed0-d7617fd3b2ed","Type":"ContainerStarted","Data":"a54d2bba5beea8ca95bf329e02f1f57884ef48bc6be2195bce62fa0ce78f8f64"} Nov 01 22:34:06.339550 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:06.339519 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:06.347468 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:06.347404 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" podStartSLOduration=1.569224596 podStartE2EDuration="5.347384556s" totalImagesPullingTime="3.77815996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:06.346318068 +0000 UTC m=+614.750908726" watchObservedRunningTime="2026-11-01 22:34:06.347384556 +0000 UTC m=+614.751975189" Nov 01 22:34:06.366366 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:06.366001 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-z9x5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 01 22:34:06.366366 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:06.366068 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" podUID="734444e5-d903-47e2-9ed0-d7617fd3b2ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 01 22:34:07.339024 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:07.338985 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-z9x5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 01 22:34:07.339425 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:07.339041 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" podUID="734444e5-d903-47e2-9ed0-d7617fd3b2ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 01 22:34:08.339895 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:08.339861 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-z9x5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 01 22:34:08.340293 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:08.339949 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" podUID="734444e5-d903-47e2-9ed0-d7617fd3b2ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 01 22:34:09.339291 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:09.339260 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-z9x5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 01 22:34:09.339487 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:09.339311 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" podUID="734444e5-d903-47e2-9ed0-d7617fd3b2ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 01 22:34:10.339481 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:10.339447 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-z9x5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 01 22:34:10.340003 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:10.339506 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" podUID="734444e5-d903-47e2-9ed0-d7617fd3b2ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 01 22:34:11.339391 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:11.339344 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-z9x5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 01 22:34:11.339600 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:11.339401 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" podUID="734444e5-d903-47e2-9ed0-d7617fd3b2ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 01 22:34:12.339187 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:12.339145 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-z9x5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 01 22:34:12.339374 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:12.339211 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" podUID="734444e5-d903-47e2-9ed0-d7617fd3b2ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 01 22:34:13.646283 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:13.646245 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:13.646747 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:13.646569 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:13.647567 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:13.647545 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" Nov 01 22:34:16.028919 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.028877 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv"] Nov 01 22:34:16.036330 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.036308 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:16.046560 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.046527 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv"] Nov 01 22:34:16.066633 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.066603 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4"] Nov 01 22:34:16.070794 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.070772 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:16.071084 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.071062 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b6bec10-9ac2-4708-9631-b76c801c0c58-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-jkvvv\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:16.071168 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.071135 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b6bec10-9ac2-4708-9631-b76c801c0c58-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-jkvvv\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:16.071168 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.071158 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-jkvvv\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:16.071257 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.071202 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b6bec10-9ac2-4708-9631-b76c801c0c58-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-jkvvv\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:16.071324 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.071311 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-jkvvv\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:16.071370 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.071359 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jkvvv\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:16.071404 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.071383 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-jkvvv\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:16.071455 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.071438 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jkvvv\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:16.071504 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.071464 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p548n\" (UniqueName: \"kubernetes.io/projected/1b6bec10-9ac2-4708-9631-b76c801c0c58-kube-api-access-p548n\") pod \"maas-default-gateway-openshift-default-6886b96f9-jkvvv\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:16.084182 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.084158 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4"] Nov 01 22:34:16.172168 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.172132 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffvbn\" (UniqueName: \"kubernetes.io/projected/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-kube-api-access-ffvbn\") pod \"maas-default-gateway-openshift-default-6886b96f9-7jkf4\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:16.172340 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.172248 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jkvvv\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:16.172340 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.172273 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-7jkf4\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:16.172340 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.172299 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-7jkf4\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:16.172439 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.172374 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b6bec10-9ac2-4708-9631-b76c801c0c58-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-jkvvv\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:16.172439 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.172408 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7jkf4\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:16.172503 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.172445 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b6bec10-9ac2-4708-9631-b76c801c0c58-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-jkvvv\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:16.172537 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.172499 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-7jkf4\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:16.172591 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.172544 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-jkvvv\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:16.172626 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.172610 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-7jkf4\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:16.172661 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.172625 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jkvvv\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:16.172694 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.172662 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jkvvv\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:16.172694 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.172685 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-jkvvv\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:16.172774 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.172707 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-7jkf4\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:16.172774 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.172738 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7jkf4\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:16.172923 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.172787 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p548n\" (UniqueName: \"kubernetes.io/projected/1b6bec10-9ac2-4708-9631-b76c801c0c58-kube-api-access-p548n\") pod \"maas-default-gateway-openshift-default-6886b96f9-jkvvv\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:16.172923 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.172817 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b6bec10-9ac2-4708-9631-b76c801c0c58-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-jkvvv\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:16.172923 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.172875 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-jkvvv\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:16.172923 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.172913 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-7jkf4\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:16.173148 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.173080 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-jkvvv\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:16.173205 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.173150 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jkvvv\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:16.173249 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.173232 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-jkvvv\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:16.173358 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.173335 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b6bec10-9ac2-4708-9631-b76c801c0c58-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-jkvvv\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:16.175569 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.175545 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b6bec10-9ac2-4708-9631-b76c801c0c58-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-jkvvv\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:16.175846 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.175811 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-jkvvv\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:16.181216 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.181197 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b6bec10-9ac2-4708-9631-b76c801c0c58-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-jkvvv\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:16.181607 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.181586 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p548n\" (UniqueName: \"kubernetes.io/projected/1b6bec10-9ac2-4708-9631-b76c801c0c58-kube-api-access-p548n\") pod \"maas-default-gateway-openshift-default-6886b96f9-jkvvv\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:16.274392 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.274362 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-7jkf4\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:16.274392 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.274394 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-7jkf4\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:16.274649 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.274416 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7jkf4\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:16.274649 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.274442 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-7jkf4\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:16.274649 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.274568 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-7jkf4\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:16.274649 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.274617 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-7jkf4\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:16.274876 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.274654 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7jkf4\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:16.274876 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.274712 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-7jkf4\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:16.274983 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.274951 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-7jkf4\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:16.275041 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.274988 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7jkf4\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:16.275041 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.275030 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffvbn\" (UniqueName: \"kubernetes.io/projected/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-kube-api-access-ffvbn\") pod \"maas-default-gateway-openshift-default-6886b96f9-7jkf4\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:16.275137 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.275049 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-7jkf4\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:16.275232 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.275213 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7jkf4\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:16.275296 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.275281 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-7jkf4\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:16.276895 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.276874 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-7jkf4\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:16.277720 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.277702 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-7jkf4\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:16.282605 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.282544 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-7jkf4\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:16.282908 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.282885 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffvbn\" (UniqueName: \"kubernetes.io/projected/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-kube-api-access-ffvbn\") pod \"maas-default-gateway-openshift-default-6886b96f9-7jkf4\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:16.351652 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.351611 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:16.384405 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.384381 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:16.631675 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.631650 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv"] Nov 01 22:34:16.633951 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:34:16.633923 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b6bec10_9ac2_4708_9631_b76c801c0c58.slice/crio-e0de9286007eba922827375bbad377993693b44f483a356c500c885c6eb8e431 WatchSource:0}: Error finding container e0de9286007eba922827375bbad377993693b44f483a356c500c885c6eb8e431: Status 404 returned error can't find the container with id e0de9286007eba922827375bbad377993693b44f483a356c500c885c6eb8e431 Nov 01 22:34:16.636950 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.636917 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:34:16.637039 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.636982 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:34:16.637039 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.637012 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:34:16.668384 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.668353 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4"] Nov 01 22:34:16.671810 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:34:16.671784 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ffbc1a2_7268_4ab3_a81c_e04b42fc441f.slice/crio-360d224eb1111977543f33e967cdd7c31a181e9a15e2f993f14567a9edf3de12 WatchSource:0}: Error finding container 360d224eb1111977543f33e967cdd7c31a181e9a15e2f993f14567a9edf3de12: Status 404 returned error can't find the container with id 360d224eb1111977543f33e967cdd7c31a181e9a15e2f993f14567a9edf3de12 Nov 01 22:34:16.674859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.674794 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:34:16.674948 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.674882 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:34:16.674948 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:16.674921 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:34:16.965435 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:34:16.965397 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:34:17.394414 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:17.394317 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" event={"ID":"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f","Type":"ContainerStarted","Data":"eae3dcb08b31ef0b28e58073fac7c78478f9a0483077bdf49648f91bed39e39d"} Nov 01 22:34:17.394414 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:17.394382 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" event={"ID":"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f","Type":"ContainerStarted","Data":"360d224eb1111977543f33e967cdd7c31a181e9a15e2f993f14567a9edf3de12"} Nov 01 22:34:17.396358 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:17.396324 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" event={"ID":"1b6bec10-9ac2-4708-9631-b76c801c0c58","Type":"ContainerStarted","Data":"f671104c94e86d233733c47eb490d4a87cdc9694df3d08517c8051a0d9f1270c"} Nov 01 22:34:17.396447 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:17.396376 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" event={"ID":"1b6bec10-9ac2-4708-9631-b76c801c0c58","Type":"ContainerStarted","Data":"e0de9286007eba922827375bbad377993693b44f483a356c500c885c6eb8e431"} Nov 01 22:34:17.416487 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:17.416442 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" podStartSLOduration=1.416430976 podStartE2EDuration="1.416430976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:17.414772322 +0000 UTC m=+625.819362981" watchObservedRunningTime="2026-11-01 22:34:17.416430976 +0000 UTC m=+625.821021607" Nov 01 22:34:17.432458 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:17.432413 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" podStartSLOduration=1.432400345 podStartE2EDuration="1.432400345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:17.431295022 +0000 UTC m=+625.835885656" watchObservedRunningTime="2026-11-01 22:34:17.432400345 +0000 UTC m=+625.836990977" Nov 01 22:34:18.361933 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:18.361895 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:18.363324 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:18.362937 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jkvvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 01 22:34:18.363324 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:18.363030 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" podUID="1b6bec10-9ac2-4708-9631-b76c801c0c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 01 22:34:18.387513 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:18.387472 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:18.387902 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:18.387880 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7jkf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 01 22:34:18.388088 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:18.388053 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" podUID="5ffbc1a2-7268-4ab3-a81c-e04b42fc441f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 01 22:34:19.360858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:19.359256 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jkvvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 01 22:34:19.360858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:19.359335 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" podUID="1b6bec10-9ac2-4708-9631-b76c801c0c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 01 22:34:19.389860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:19.386301 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7jkf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 01 22:34:19.389860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:19.386377 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" podUID="5ffbc1a2-7268-4ab3-a81c-e04b42fc441f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 01 22:34:20.353372 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:20.353337 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jkvvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 01 22:34:20.353695 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:20.353651 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" podUID="1b6bec10-9ac2-4708-9631-b76c801c0c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 01 22:34:20.385856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:20.385288 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7jkf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 01 22:34:20.385856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:20.385366 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" podUID="5ffbc1a2-7268-4ab3-a81c-e04b42fc441f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 01 22:34:21.361931 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:21.361255 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jkvvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 01 22:34:21.361931 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:21.361323 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" podUID="1b6bec10-9ac2-4708-9631-b76c801c0c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 01 22:34:21.386867 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:21.386409 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7jkf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 01 22:34:21.386867 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:21.386468 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" podUID="5ffbc1a2-7268-4ab3-a81c-e04b42fc441f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 01 22:34:22.353298 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:22.353258 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jkvvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 01 22:34:22.353587 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:22.353553 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" podUID="1b6bec10-9ac2-4708-9631-b76c801c0c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 01 22:34:22.387859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:22.387303 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7jkf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 01 22:34:22.387859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:22.387380 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" podUID="5ffbc1a2-7268-4ab3-a81c-e04b42fc441f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 01 22:34:23.352624 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:23.352588 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jkvvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 01 22:34:23.352814 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:23.352650 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" podUID="1b6bec10-9ac2-4708-9631-b76c801c0c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 01 22:34:23.385818 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:23.385781 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7jkf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 01 22:34:23.386024 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:23.385865 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" podUID="5ffbc1a2-7268-4ab3-a81c-e04b42fc441f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 01 22:34:25.011513 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:25.011477 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:25.012811 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:25.012750 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:25.019364 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:25.019338 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 22:34:25.067635 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:25.067578 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:25.068803 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:25.068779 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:25.069915 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:25.069887 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:34:46.959280 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:34:46.959202 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:34:52.503560 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.503527 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7"] Nov 01 22:34:52.512174 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.512145 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:52.519572 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.519518 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7"] Nov 01 22:34:52.521203 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.521007 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-e26733-openshift-default-dockercfg-794k8\"" Nov 01 22:34:52.657644 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.657602 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-workload-certs\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:52.657842 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.657674 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0d043d9-49a8-42fb-872c-cd5479590b1f-istiod-ca-cert\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:52.657901 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.657843 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-workload-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:52.657901 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.657872 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-credential-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:52.657901 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.657896 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0d043d9-49a8-42fb-872c-cd5479590b1f-istio-podinfo\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:52.658055 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.657916 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-istio-data\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:52.658055 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.658030 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-istio-envoy\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:52.658055 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.658049 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9smxt\" (UniqueName: \"kubernetes.io/projected/b0d043d9-49a8-42fb-872c-cd5479590b1f-kube-api-access-9smxt\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:52.658170 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.658068 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0d043d9-49a8-42fb-872c-cd5479590b1f-istio-token\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:52.749824 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.749792 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf"] Nov 01 22:34:52.754768 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.754709 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:52.759022 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.758997 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0d043d9-49a8-42fb-872c-cd5479590b1f-istiod-ca-cert\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:52.759157 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.759041 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-workload-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:52.759157 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.759058 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-credential-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:52.759157 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.759079 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0d043d9-49a8-42fb-872c-cd5479590b1f-istio-podinfo\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:52.759157 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.759101 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-istio-data\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:52.759157 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.759139 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-istio-envoy\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:52.759421 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.759169 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9smxt\" (UniqueName: \"kubernetes.io/projected/b0d043d9-49a8-42fb-872c-cd5479590b1f-kube-api-access-9smxt\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:52.759421 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.759199 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0d043d9-49a8-42fb-872c-cd5479590b1f-istio-token\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:52.759421 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.759256 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-workload-certs\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:52.759585 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.759533 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-workload-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:52.759885 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.759863 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-workload-certs\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:52.760054 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.759875 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-credential-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:52.760175 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.759943 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-istio-data\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:52.760239 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.760172 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0d043d9-49a8-42fb-872c-cd5479590b1f-istiod-ca-cert\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:52.761574 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.761551 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-istio-envoy\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:52.762041 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.762014 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0d043d9-49a8-42fb-872c-cd5479590b1f-istio-podinfo\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:52.763673 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.763646 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-e5f47e-openshift-default-dockercfg-lfvrz\"" Nov 01 22:34:52.770723 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.770701 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf"] Nov 01 22:34:52.792282 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.792251 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0d043d9-49a8-42fb-872c-cd5479590b1f-istio-token\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:52.793801 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.793778 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9smxt\" (UniqueName: \"kubernetes.io/projected/b0d043d9-49a8-42fb-872c-cd5479590b1f-kube-api-access-9smxt\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:52.831382 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.831352 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:52.860207 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.860176 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istio-data\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:52.860363 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.860216 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istiod-ca-cert\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:52.860363 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.860253 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq655\" (UniqueName: \"kubernetes.io/projected/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-kube-api-access-sq655\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:52.860630 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.860601 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istio-token\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:52.860844 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.860795 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-workload-certs\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:52.861002 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.860914 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-credential-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:52.861002 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.860974 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-workload-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:52.861119 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.861103 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istio-envoy\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:52.861187 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.861138 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istio-podinfo\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:52.961846 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.961784 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istio-token\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:52.962041 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.961865 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-workload-certs\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:52.962041 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.961917 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-credential-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:52.962041 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.961961 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-workload-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:52.962041 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.962006 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istio-envoy\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:52.962041 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.962025 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istio-podinfo\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:52.962315 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.962050 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istio-data\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:52.962315 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.962069 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istiod-ca-cert\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:52.962315 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.962100 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq655\" (UniqueName: \"kubernetes.io/projected/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-kube-api-access-sq655\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:52.962478 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.962322 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-workload-certs\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:52.962608 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.962588 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istio-data\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:52.963137 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.963109 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-credential-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:52.963271 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.963197 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istiod-ca-cert\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:52.963389 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.963366 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-workload-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:52.964438 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.964420 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istio-envoy\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:52.964697 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.964678 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istio-podinfo\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:52.981860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.981740 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istio-token\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:52.981860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:52.981798 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq655\" (UniqueName: \"kubernetes.io/projected/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-kube-api-access-sq655\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:53.084181 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.084095 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:53.184798 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:34:53.184771 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d043d9_49a8_42fb_872c_cd5479590b1f.slice/crio-413ba639462ecdb6ec20a55deec3a17e84c39ccadb1d20d1fe50c537e01fa053 WatchSource:0}: Error finding container 413ba639462ecdb6ec20a55deec3a17e84c39ccadb1d20d1fe50c537e01fa053: Status 404 returned error can't find the container with id 413ba639462ecdb6ec20a55deec3a17e84c39ccadb1d20d1fe50c537e01fa053 Nov 01 22:34:53.187696 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.187662 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:34:53.187789 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.187742 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:34:53.187871 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.187790 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:34:53.193143 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.193112 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7"] Nov 01 22:34:53.203819 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.203675 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx"] Nov 01 22:34:53.208971 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.208950 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:53.263043 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.252986 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx"] Nov 01 22:34:53.271244 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.271202 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-6ac182-openshift-default-dockercfg-mf2q8\"" Nov 01 22:34:53.366320 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.366216 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-istio-data\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:53.366320 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.366293 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d4e19a-0aea-4d86-9876-23181ac12a4a-istiod-ca-cert\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:53.366538 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.366324 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d4e19a-0aea-4d86-9876-23181ac12a4a-istio-podinfo\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:53.366538 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.366350 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d4e19a-0aea-4d86-9876-23181ac12a4a-istio-token\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:53.366538 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.366391 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzt88\" (UniqueName: \"kubernetes.io/projected/02d4e19a-0aea-4d86-9876-23181ac12a4a-kube-api-access-lzt88\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:53.366538 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.366472 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-workload-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:53.366815 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.366548 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-istio-envoy\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:53.366815 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.366584 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-workload-certs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:53.366815 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.366617 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-credential-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:53.397772 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.397739 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf"] Nov 01 22:34:53.398224 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:34:53.398199 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5730c77_aecb_441e_82f5_9d34e5e3b8d5.slice/crio-61c186f25ca675982590e8641f3571e42d528da60e286875a2dd3c973abf3dc0 WatchSource:0}: Error finding container 61c186f25ca675982590e8641f3571e42d528da60e286875a2dd3c973abf3dc0: Status 404 returned error can't find the container with id 61c186f25ca675982590e8641f3571e42d528da60e286875a2dd3c973abf3dc0 Nov 01 22:34:53.400476 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.400440 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:34:53.400566 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.400510 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:34:53.400566 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.400541 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:34:53.467496 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.467463 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d4e19a-0aea-4d86-9876-23181ac12a4a-istio-podinfo\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:53.467852 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.467516 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d4e19a-0aea-4d86-9876-23181ac12a4a-istio-token\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:53.467852 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.467568 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzt88\" (UniqueName: \"kubernetes.io/projected/02d4e19a-0aea-4d86-9876-23181ac12a4a-kube-api-access-lzt88\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:53.467852 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.467644 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-workload-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:53.467852 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.467720 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-istio-envoy\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:53.467852 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.467748 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-workload-certs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:53.467852 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.467778 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-credential-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:53.467852 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.467804 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-istio-data\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:53.468260 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.467895 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d4e19a-0aea-4d86-9876-23181ac12a4a-istiod-ca-cert\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:53.468322 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.468295 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-workload-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:53.468736 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.468708 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d4e19a-0aea-4d86-9876-23181ac12a4a-istiod-ca-cert\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:53.469087 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.468920 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-credential-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:53.469087 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.468990 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-workload-certs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:53.469238 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.469112 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-istio-data\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:53.471716 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.471676 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-istio-envoy\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:53.472066 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.472050 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d4e19a-0aea-4d86-9876-23181ac12a4a-istio-podinfo\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:53.494984 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.494951 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d4e19a-0aea-4d86-9876-23181ac12a4a-istio-token\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:53.495318 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.495295 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzt88\" (UniqueName: \"kubernetes.io/projected/02d4e19a-0aea-4d86-9876-23181ac12a4a-kube-api-access-lzt88\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:53.576275 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.576246 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:53.588326 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.588275 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" event={"ID":"b0d043d9-49a8-42fb-872c-cd5479590b1f","Type":"ContainerStarted","Data":"3d39ef4a83cc4951b1891e0c772456f2b9022de0303a5895c61b9e1bb550f43d"} Nov 01 22:34:53.588634 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.588606 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" event={"ID":"b0d043d9-49a8-42fb-872c-cd5479590b1f","Type":"ContainerStarted","Data":"413ba639462ecdb6ec20a55deec3a17e84c39ccadb1d20d1fe50c537e01fa053"} Nov 01 22:34:53.593997 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.592045 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" event={"ID":"b5730c77-aecb-441e-82f5-9d34e5e3b8d5","Type":"ContainerStarted","Data":"444caa99c6b01836eaa1d41f577683c7cf47b49f05483f5bc886609c980aab3c"} Nov 01 22:34:53.593997 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.592094 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" event={"ID":"b5730c77-aecb-441e-82f5-9d34e5e3b8d5","Type":"ContainerStarted","Data":"61c186f25ca675982590e8641f3571e42d528da60e286875a2dd3c973abf3dc0"} Nov 01 22:34:53.627896 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.627766 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" podStartSLOduration=1.627747482 podStartE2EDuration="1.627747482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:53.625610997 +0000 UTC m=+662.030201684" watchObservedRunningTime="2026-11-01 22:34:53.627747482 +0000 UTC m=+662.032338116" Nov 01 22:34:53.684867 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.684722 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" podStartSLOduration=1.6847074869999998 podStartE2EDuration="1.684707487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:53.677494419 +0000 UTC m=+662.082085053" watchObservedRunningTime="2026-11-01 22:34:53.684707487 +0000 UTC m=+662.089298119" Nov 01 22:34:53.833880 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.833640 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:53.922448 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.922422 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx"] Nov 01 22:34:53.924821 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:34:53.924794 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02d4e19a_0aea_4d86_9876_23181ac12a4a.slice/crio-f884a9904d9b1fddcf9526185f9031d9e799e43c913d40d85b379e5d1ce707ef WatchSource:0}: Error finding container f884a9904d9b1fddcf9526185f9031d9e799e43c913d40d85b379e5d1ce707ef: Status 404 returned error can't find the container with id f884a9904d9b1fddcf9526185f9031d9e799e43c913d40d85b379e5d1ce707ef Nov 01 22:34:53.927979 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.927937 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:34:53.928097 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.928010 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:34:53.928097 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:53.928044 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:34:54.084327 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:54.084289 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:54.090587 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:54.090339 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:54.599292 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:54.599250 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" event={"ID":"02d4e19a-0aea-4d86-9876-23181ac12a4a","Type":"ContainerStarted","Data":"a9c4ce39758d1535eabde8e9d66a55bc4feeed6d5022437cdcbf75a485cb5887"} Nov 01 22:34:54.599789 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:54.599305 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" event={"ID":"02d4e19a-0aea-4d86-9876-23181ac12a4a","Type":"ContainerStarted","Data":"f884a9904d9b1fddcf9526185f9031d9e799e43c913d40d85b379e5d1ce707ef"} Nov 01 22:34:54.599789 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:54.599655 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:54.600368 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:54.600352 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:34:54.620503 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:54.620451 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" podStartSLOduration=1.620436607 podStartE2EDuration="1.620436607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:54.618731242 +0000 UTC m=+663.023321874" watchObservedRunningTime="2026-11-01 22:34:54.620436607 +0000 UTC m=+663.025027239" Nov 01 22:34:54.851102 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:54.850974 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:55.577375 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:55.577326 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:55.582691 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:55.582663 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:55.605812 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:55.605781 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:55.606259 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:55.605873 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:55.606560 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:55.606540 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:34:55.606764 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:55.606749 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:34:57.978946 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:57.978907 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2f3705-855984bb47-ztmpt"] Nov 01 22:34:57.982977 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:57.982953 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2f3705-855984bb47-ztmpt" Nov 01 22:34:57.994270 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:57.994244 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-2f3705\"" Nov 01 22:34:57.999122 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:57.999092 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2f3705-855984bb47-ztmpt"] Nov 01 22:34:58.020930 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:58.020890 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-2f3705-855984bb47-ztmpt\" (UID: \"c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2f3705-855984bb47-ztmpt" Nov 01 22:34:58.021108 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:58.020953 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw5lh\" (UniqueName: \"kubernetes.io/projected/c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583-kube-api-access-mw5lh\") pod \"payload-pre-processing-e2e-worker-w4-2f3705-855984bb47-ztmpt\" (UID: \"c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2f3705-855984bb47-ztmpt" Nov 01 22:34:58.045645 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:58.045609 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2f3705-64d47ccbdf-xgv67"] Nov 01 22:34:58.049294 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:58.049275 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-2f3705-64d47ccbdf-xgv67" Nov 01 22:34:58.059113 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:58.059093 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-2f3705-dockercfg-8h84t\"" Nov 01 22:34:58.064661 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:58.064637 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2f3705-64d47ccbdf-xgv67"] Nov 01 22:34:58.131856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:58.126956 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhwwm\" (UniqueName: \"kubernetes.io/projected/0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce-kube-api-access-nhwwm\") pod \"payload-processing-e2e-worker-w4-2f3705-64d47ccbdf-xgv67\" (UID: \"0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2f3705-64d47ccbdf-xgv67" Nov 01 22:34:58.131856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:58.127116 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-2f3705-855984bb47-ztmpt\" (UID: \"c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2f3705-855984bb47-ztmpt" Nov 01 22:34:58.131856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:58.127167 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-2f3705-64d47ccbdf-xgv67\" (UID: \"0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2f3705-64d47ccbdf-xgv67" Nov 01 22:34:58.131856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:58.127200 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw5lh\" (UniqueName: \"kubernetes.io/projected/c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583-kube-api-access-mw5lh\") pod \"payload-pre-processing-e2e-worker-w4-2f3705-855984bb47-ztmpt\" (UID: \"c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2f3705-855984bb47-ztmpt" Nov 01 22:34:58.131856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:58.128408 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-2f3705-855984bb47-ztmpt\" (UID: \"c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2f3705-855984bb47-ztmpt" Nov 01 22:34:58.165895 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:58.165242 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw5lh\" (UniqueName: \"kubernetes.io/projected/c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583-kube-api-access-mw5lh\") pod \"payload-pre-processing-e2e-worker-w4-2f3705-855984bb47-ztmpt\" (UID: \"c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2f3705-855984bb47-ztmpt" Nov 01 22:34:58.228422 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:58.228389 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhwwm\" (UniqueName: \"kubernetes.io/projected/0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce-kube-api-access-nhwwm\") pod \"payload-processing-e2e-worker-w4-2f3705-64d47ccbdf-xgv67\" (UID: \"0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2f3705-64d47ccbdf-xgv67" Nov 01 22:34:58.228633 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:58.228476 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-2f3705-64d47ccbdf-xgv67\" (UID: \"0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2f3705-64d47ccbdf-xgv67" Nov 01 22:34:58.229307 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:58.229217 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-2f3705-64d47ccbdf-xgv67\" (UID: \"0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2f3705-64d47ccbdf-xgv67" Nov 01 22:34:58.237944 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:58.237910 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhwwm\" (UniqueName: \"kubernetes.io/projected/0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce-kube-api-access-nhwwm\") pod \"payload-processing-e2e-worker-w4-2f3705-64d47ccbdf-xgv67\" (UID: \"0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2f3705-64d47ccbdf-xgv67" Nov 01 22:34:58.296182 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:58.296149 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2f3705-855984bb47-ztmpt" Nov 01 22:34:58.365474 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:58.363222 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-2f3705-64d47ccbdf-xgv67" Nov 01 22:34:58.577239 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:58.577202 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2f3705-855984bb47-ztmpt"] Nov 01 22:34:58.579480 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:34:58.579456 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5cfa8f2_966b_4b28_b8ff_0c23d4fd2583.slice/crio-7573b4c739245a5dfdefeedfcd836ca5e69810890b0a08a76c110566c88b8585 WatchSource:0}: Error finding container 7573b4c739245a5dfdefeedfcd836ca5e69810890b0a08a76c110566c88b8585: Status 404 returned error can't find the container with id 7573b4c739245a5dfdefeedfcd836ca5e69810890b0a08a76c110566c88b8585 Nov 01 22:34:58.581255 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:58.581233 5011 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:34:58.619535 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:58.619501 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2f3705-855984bb47-ztmpt" event={"ID":"c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583","Type":"ContainerStarted","Data":"7573b4c739245a5dfdefeedfcd836ca5e69810890b0a08a76c110566c88b8585"} Nov 01 22:34:58.660713 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:58.660688 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2f3705-64d47ccbdf-xgv67"] Nov 01 22:34:58.661440 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:34:58.661413 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e8a8d7b_bc08_4d30_a435_dfd4d4ed76ce.slice/crio-78ae3d827fc6d07a9b60aee15484cea0e4cc51a88e26d4dbb0021e5126f25fb2 WatchSource:0}: Error finding container 78ae3d827fc6d07a9b60aee15484cea0e4cc51a88e26d4dbb0021e5126f25fb2: Status 404 returned error can't find the container with id 78ae3d827fc6d07a9b60aee15484cea0e4cc51a88e26d4dbb0021e5126f25fb2 Nov 01 22:34:59.629301 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:59.629231 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-2f3705-64d47ccbdf-xgv67" event={"ID":"0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce","Type":"ContainerStarted","Data":"78ae3d827fc6d07a9b60aee15484cea0e4cc51a88e26d4dbb0021e5126f25fb2"} Nov 01 22:34:59.654375 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:59.652520 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e5f47e-6b9bd944c5-jbtnp"] Nov 01 22:34:59.656644 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:59.656623 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e5f47e-6b9bd944c5-jbtnp" Nov 01 22:34:59.675121 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:59.675079 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e5f47e-6b9bd944c5-jbtnp"] Nov 01 22:34:59.744895 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:59.726129 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-e5f47e\"" Nov 01 22:34:59.744895 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:59.726268 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-e5f47e-dockercfg-xhwjj\"" Nov 01 22:34:59.770887 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:59.756907 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqblr\" (UniqueName: \"kubernetes.io/projected/fffb686c-2a8e-469c-bdd5-7eeb4b03bc06-kube-api-access-dqblr\") pod \"payload-processing-e2e-worker-w0-e5f47e-6b9bd944c5-jbtnp\" (UID: \"fffb686c-2a8e-469c-bdd5-7eeb4b03bc06\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e5f47e-6b9bd944c5-jbtnp" Nov 01 22:34:59.770887 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:59.756973 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fffb686c-2a8e-469c-bdd5-7eeb4b03bc06-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-e5f47e-6b9bd944c5-jbtnp\" (UID: \"fffb686c-2a8e-469c-bdd5-7eeb4b03bc06\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e5f47e-6b9bd944c5-jbtnp" Nov 01 22:34:59.854234 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:59.854204 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5"] Nov 01 22:34:59.858413 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:59.858383 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqblr\" (UniqueName: \"kubernetes.io/projected/fffb686c-2a8e-469c-bdd5-7eeb4b03bc06-kube-api-access-dqblr\") pod \"payload-processing-e2e-worker-w0-e5f47e-6b9bd944c5-jbtnp\" (UID: \"fffb686c-2a8e-469c-bdd5-7eeb4b03bc06\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e5f47e-6b9bd944c5-jbtnp" Nov 01 22:34:59.858413 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:59.858457 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fffb686c-2a8e-469c-bdd5-7eeb4b03bc06-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-e5f47e-6b9bd944c5-jbtnp\" (UID: \"fffb686c-2a8e-469c-bdd5-7eeb4b03bc06\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e5f47e-6b9bd944c5-jbtnp" Nov 01 22:34:59.859322 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:59.859274 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fffb686c-2a8e-469c-bdd5-7eeb4b03bc06-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-e5f47e-6b9bd944c5-jbtnp\" (UID: \"fffb686c-2a8e-469c-bdd5-7eeb4b03bc06\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e5f47e-6b9bd944c5-jbtnp" Nov 01 22:34:59.860720 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:59.860002 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5" Nov 01 22:34:59.877683 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:59.877626 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5"] Nov 01 22:34:59.879372 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:59.877658 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:34:59.879507 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:59.877966 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:34:59.881306 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:59.881254 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqblr\" (UniqueName: \"kubernetes.io/projected/fffb686c-2a8e-469c-bdd5-7eeb4b03bc06-kube-api-access-dqblr\") pod \"payload-processing-e2e-worker-w0-e5f47e-6b9bd944c5-jbtnp\" (UID: \"fffb686c-2a8e-469c-bdd5-7eeb4b03bc06\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e5f47e-6b9bd944c5-jbtnp" Nov 01 22:34:59.919872 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:59.919585 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-2f3705-serving-cert\"" Nov 01 22:34:59.919872 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:59.919644 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-57phm\"" Nov 01 22:34:59.920149 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:59.920125 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 22:34:59.960493 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:59.960450 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91254bf5-2625-4e5b-bbf1-c802b51b8438-maas-api-tls\") pod \"maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5\" (UID: \"91254bf5-2625-4e5b-bbf1-c802b51b8438\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5" Nov 01 22:34:59.960674 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:59.960539 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gc94\" (UniqueName: \"kubernetes.io/projected/91254bf5-2625-4e5b-bbf1-c802b51b8438-kube-api-access-7gc94\") pod \"maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5\" (UID: \"91254bf5-2625-4e5b-bbf1-c802b51b8438\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5" Nov 01 22:34:59.960674 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:59.960617 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91254bf5-2625-4e5b-bbf1-c802b51b8438-metrics-certs\") pod \"maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5\" (UID: \"91254bf5-2625-4e5b-bbf1-c802b51b8438\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5" Nov 01 22:34:59.960904 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:59.960821 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91254bf5-2625-4e5b-bbf1-c802b51b8438-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5\" (UID: \"91254bf5-2625-4e5b-bbf1-c802b51b8438\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5" Nov 01 22:34:59.980267 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:34:59.980232 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e5f47e-6b9bd944c5-jbtnp" Nov 01 22:35:00.065198 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:00.061917 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91254bf5-2625-4e5b-bbf1-c802b51b8438-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5\" (UID: \"91254bf5-2625-4e5b-bbf1-c802b51b8438\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5" Nov 01 22:35:00.065198 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:00.061989 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91254bf5-2625-4e5b-bbf1-c802b51b8438-maas-api-tls\") pod \"maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5\" (UID: \"91254bf5-2625-4e5b-bbf1-c802b51b8438\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5" Nov 01 22:35:00.067786 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:00.065585 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gc94\" (UniqueName: \"kubernetes.io/projected/91254bf5-2625-4e5b-bbf1-c802b51b8438-kube-api-access-7gc94\") pod \"maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5\" (UID: \"91254bf5-2625-4e5b-bbf1-c802b51b8438\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5" Nov 01 22:35:00.067968 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:00.067897 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91254bf5-2625-4e5b-bbf1-c802b51b8438-metrics-certs\") pod \"maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5\" (UID: \"91254bf5-2625-4e5b-bbf1-c802b51b8438\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5" Nov 01 22:35:00.068359 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:00.067516 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91254bf5-2625-4e5b-bbf1-c802b51b8438-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5\" (UID: \"91254bf5-2625-4e5b-bbf1-c802b51b8438\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5" Nov 01 22:35:00.074939 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:00.074440 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91254bf5-2625-4e5b-bbf1-c802b51b8438-maas-api-tls\") pod \"maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5\" (UID: \"91254bf5-2625-4e5b-bbf1-c802b51b8438\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5" Nov 01 22:35:00.075943 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:00.075590 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91254bf5-2625-4e5b-bbf1-c802b51b8438-metrics-certs\") pod \"maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5\" (UID: \"91254bf5-2625-4e5b-bbf1-c802b51b8438\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5" Nov 01 22:35:00.087554 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:00.087522 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gc94\" (UniqueName: \"kubernetes.io/projected/91254bf5-2625-4e5b-bbf1-c802b51b8438-kube-api-access-7gc94\") pod \"maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5\" (UID: \"91254bf5-2625-4e5b-bbf1-c802b51b8438\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5" Nov 01 22:35:00.185129 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:00.185086 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5" Nov 01 22:35:00.317014 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:00.316968 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e5f47e-6b9bd944c5-jbtnp"] Nov 01 22:35:00.319965 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:35:00.319932 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfffb686c_2a8e_469c_bdd5_7eeb4b03bc06.slice/crio-09ddcebd51d852577956c92c6704bc6db6ce184e609581d6d891b3fa9cb5ebb1 WatchSource:0}: Error finding container 09ddcebd51d852577956c92c6704bc6db6ce184e609581d6d891b3fa9cb5ebb1: Status 404 returned error can't find the container with id 09ddcebd51d852577956c92c6704bc6db6ce184e609581d6d891b3fa9cb5ebb1 Nov 01 22:35:00.535741 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:00.535652 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5"] Nov 01 22:35:00.546000 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:35:00.545961 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91254bf5_2625_4e5b_bbf1_c802b51b8438.slice/crio-807475c0de97b9f7fb3e7390d88b8f87f6d0105d0cb4c19f3e0830ed18de8aa8 WatchSource:0}: Error finding container 807475c0de97b9f7fb3e7390d88b8f87f6d0105d0cb4c19f3e0830ed18de8aa8: Status 404 returned error can't find the container with id 807475c0de97b9f7fb3e7390d88b8f87f6d0105d0cb4c19f3e0830ed18de8aa8 Nov 01 22:35:00.649222 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:00.649166 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5" event={"ID":"91254bf5-2625-4e5b-bbf1-c802b51b8438","Type":"ContainerStarted","Data":"807475c0de97b9f7fb3e7390d88b8f87f6d0105d0cb4c19f3e0830ed18de8aa8"} Nov 01 22:35:00.650757 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:00.650726 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e5f47e-6b9bd944c5-jbtnp" event={"ID":"fffb686c-2a8e-469c-bdd5-7eeb4b03bc06","Type":"ContainerStarted","Data":"09ddcebd51d852577956c92c6704bc6db6ce184e609581d6d891b3fa9cb5ebb1"} Nov 01 22:35:01.666903 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:01.666862 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5" event={"ID":"91254bf5-2625-4e5b-bbf1-c802b51b8438","Type":"ContainerStarted","Data":"eec05dce6f13cb05a05683516241673e24a875e90182bf72bb9e71eeede71eb8"} Nov 01 22:35:01.667404 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:01.667074 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5" Nov 01 22:35:02.085068 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.083976 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5" podStartSLOduration=2.55734687 podStartE2EDuration="3.083961793s" totalImagesPullingTime="526.614923ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:01.703681988 +0000 UTC m=+670.108272644" watchObservedRunningTime="2026-11-01 22:35:02.083961793 +0000 UTC m=+670.488552457" Nov 01 22:35:02.086399 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.086348 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs"] Nov 01 22:35:02.090726 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.090686 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs" Nov 01 22:35:02.101114 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.101085 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs"] Nov 01 22:35:02.103805 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.103780 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-6ac182-dockercfg-fdj7k\"" Nov 01 22:35:02.103961 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.103942 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-6ac182\"" Nov 01 22:35:02.310964 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.310735 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af426678-7013-404d-a85c-555c7abba8c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs\" (UID: \"af426678-7013-404d-a85c-555c7abba8c7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs" Nov 01 22:35:02.310964 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.310808 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rqxx\" (UniqueName: \"kubernetes.io/projected/af426678-7013-404d-a85c-555c7abba8c7-kube-api-access-8rqxx\") pod \"payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs\" (UID: \"af426678-7013-404d-a85c-555c7abba8c7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs" Nov 01 22:35:02.412163 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.412126 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af426678-7013-404d-a85c-555c7abba8c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs\" (UID: \"af426678-7013-404d-a85c-555c7abba8c7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs" Nov 01 22:35:02.412333 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.412181 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rqxx\" (UniqueName: \"kubernetes.io/projected/af426678-7013-404d-a85c-555c7abba8c7-kube-api-access-8rqxx\") pod \"payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs\" (UID: \"af426678-7013-404d-a85c-555c7abba8c7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs" Nov 01 22:35:02.412860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.412825 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af426678-7013-404d-a85c-555c7abba8c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs\" (UID: \"af426678-7013-404d-a85c-555c7abba8c7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs" Nov 01 22:35:02.422687 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.422655 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rqxx\" (UniqueName: \"kubernetes.io/projected/af426678-7013-404d-a85c-555c7abba8c7-kube-api-access-8rqxx\") pod \"payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs\" (UID: \"af426678-7013-404d-a85c-555c7abba8c7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs" Nov 01 22:35:02.608629 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.608545 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk"] Nov 01 22:35:02.612937 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.612911 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk" Nov 01 22:35:02.618738 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.618707 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-e5f47e-serving-cert\"" Nov 01 22:35:02.622100 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.622061 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk"] Nov 01 22:35:02.673651 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.673603 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e5f47e-6b9bd944c5-jbtnp" event={"ID":"fffb686c-2a8e-469c-bdd5-7eeb4b03bc06","Type":"ContainerStarted","Data":"e544a092bee694d9400bd257e2d9a5b8a54bca2e2e1a488c37eeb46d9558f8dc"} Nov 01 22:35:02.674127 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.673754 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-e5f47e-6b9bd944c5-jbtnp" Nov 01 22:35:02.676379 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.676349 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2f3705-855984bb47-ztmpt" event={"ID":"c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583","Type":"ContainerStarted","Data":"3e841d9918502a08128a3d7cbf9ee34e3cef12fb39a1bd5651109c7d1cc9d9c9"} Nov 01 22:35:02.676530 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.676405 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2f3705-855984bb47-ztmpt" Nov 01 22:35:02.678338 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.678311 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-2f3705-64d47ccbdf-xgv67" event={"ID":"0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce","Type":"ContainerStarted","Data":"e6677851f8cddef76b5db75ac3e4d1e306603fb21b97ae112df858390f784972"} Nov 01 22:35:02.678460 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.678418 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-2f3705-64d47ccbdf-xgv67" Nov 01 22:35:02.696808 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.696756 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-e5f47e-6b9bd944c5-jbtnp" podStartSLOduration=1.562112254 podStartE2EDuration="3.696741749s" totalImagesPullingTime="2.134629495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:02.695689525 +0000 UTC m=+671.100280159" watchObservedRunningTime="2026-11-01 22:35:02.696741749 +0000 UTC m=+671.101332390" Nov 01 22:35:02.714716 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.714682 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs" Nov 01 22:35:02.715113 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.715093 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abea67e1-7f6b-4abd-a1f1-13a2b1613458-metrics-certs\") pod \"maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk\" (UID: \"abea67e1-7f6b-4abd-a1f1-13a2b1613458\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk" Nov 01 22:35:02.715417 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.715383 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abea67e1-7f6b-4abd-a1f1-13a2b1613458-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk\" (UID: \"abea67e1-7f6b-4abd-a1f1-13a2b1613458\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk" Nov 01 22:35:02.715528 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.715492 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abea67e1-7f6b-4abd-a1f1-13a2b1613458-maas-api-tls\") pod \"maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk\" (UID: \"abea67e1-7f6b-4abd-a1f1-13a2b1613458\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk" Nov 01 22:35:02.715581 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.715525 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d2zs\" (UniqueName: \"kubernetes.io/projected/abea67e1-7f6b-4abd-a1f1-13a2b1613458-kube-api-access-5d2zs\") pod \"maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk\" (UID: \"abea67e1-7f6b-4abd-a1f1-13a2b1613458\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk" Nov 01 22:35:02.721316 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.721268 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2f3705-855984bb47-ztmpt" podStartSLOduration=2.5518407659999998 podStartE2EDuration="5.721254847s" totalImagesPullingTime="3.169414081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:02.719418458 +0000 UTC m=+671.124009094" watchObservedRunningTime="2026-11-01 22:35:02.721254847 +0000 UTC m=+671.125845478" Nov 01 22:35:02.748736 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.748675 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-2f3705-64d47ccbdf-xgv67" podStartSLOduration=1.488862768 podStartE2EDuration="4.748657434s" totalImagesPullingTime="3.259794666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:02.746294306 +0000 UTC m=+671.150884938" watchObservedRunningTime="2026-11-01 22:35:02.748657434 +0000 UTC m=+671.153248068" Nov 01 22:35:02.816578 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.816538 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abea67e1-7f6b-4abd-a1f1-13a2b1613458-maas-api-tls\") pod \"maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk\" (UID: \"abea67e1-7f6b-4abd-a1f1-13a2b1613458\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk" Nov 01 22:35:02.816578 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.816580 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d2zs\" (UniqueName: \"kubernetes.io/projected/abea67e1-7f6b-4abd-a1f1-13a2b1613458-kube-api-access-5d2zs\") pod \"maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk\" (UID: \"abea67e1-7f6b-4abd-a1f1-13a2b1613458\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk" Nov 01 22:35:02.816825 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.816665 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abea67e1-7f6b-4abd-a1f1-13a2b1613458-metrics-certs\") pod \"maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk\" (UID: \"abea67e1-7f6b-4abd-a1f1-13a2b1613458\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk" Nov 01 22:35:02.816825 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.816786 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abea67e1-7f6b-4abd-a1f1-13a2b1613458-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk\" (UID: \"abea67e1-7f6b-4abd-a1f1-13a2b1613458\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk" Nov 01 22:35:02.817528 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.817448 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abea67e1-7f6b-4abd-a1f1-13a2b1613458-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk\" (UID: \"abea67e1-7f6b-4abd-a1f1-13a2b1613458\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk" Nov 01 22:35:02.819519 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.819489 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abea67e1-7f6b-4abd-a1f1-13a2b1613458-metrics-certs\") pod \"maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk\" (UID: \"abea67e1-7f6b-4abd-a1f1-13a2b1613458\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk" Nov 01 22:35:02.819625 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.819523 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abea67e1-7f6b-4abd-a1f1-13a2b1613458-maas-api-tls\") pod \"maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk\" (UID: \"abea67e1-7f6b-4abd-a1f1-13a2b1613458\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk" Nov 01 22:35:02.832033 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.832012 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d2zs\" (UniqueName: \"kubernetes.io/projected/abea67e1-7f6b-4abd-a1f1-13a2b1613458-kube-api-access-5d2zs\") pod \"maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk\" (UID: \"abea67e1-7f6b-4abd-a1f1-13a2b1613458\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk" Nov 01 22:35:02.935025 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:02.934996 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk" Nov 01 22:35:03.051056 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:03.045891 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs"] Nov 01 22:35:03.057385 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:35:03.057329 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf426678_7013_404d_a85c_555c7abba8c7.slice/crio-208fdc1d286b6fcd172ca3d9087be4919cb90eb69801504114c09923862ff6af WatchSource:0}: Error finding container 208fdc1d286b6fcd172ca3d9087be4919cb90eb69801504114c09923862ff6af: Status 404 returned error can't find the container with id 208fdc1d286b6fcd172ca3d9087be4919cb90eb69801504114c09923862ff6af Nov 01 22:35:03.283981 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:03.283890 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk"] Nov 01 22:35:03.289193 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:35:03.289162 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabea67e1_7f6b_4abd_a1f1_13a2b1613458.slice/crio-18f4fa6a44e12e0bbf1fbf1465246362e55354474f36832eb3030668e3e31331 WatchSource:0}: Error finding container 18f4fa6a44e12e0bbf1fbf1465246362e55354474f36832eb3030668e3e31331: Status 404 returned error can't find the container with id 18f4fa6a44e12e0bbf1fbf1465246362e55354474f36832eb3030668e3e31331 Nov 01 22:35:03.685913 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:03.685876 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk" event={"ID":"abea67e1-7f6b-4abd-a1f1-13a2b1613458","Type":"ContainerStarted","Data":"18f4fa6a44e12e0bbf1fbf1465246362e55354474f36832eb3030668e3e31331"} Nov 01 22:35:03.688916 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:03.688188 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs" event={"ID":"af426678-7013-404d-a85c-555c7abba8c7","Type":"ContainerStarted","Data":"c3611b76aac7926b7ff46797106c1d74fcfac16be77ac022e876f5ff6c4bc880"} Nov 01 22:35:03.688916 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:03.688249 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs" event={"ID":"af426678-7013-404d-a85c-555c7abba8c7","Type":"ContainerStarted","Data":"208fdc1d286b6fcd172ca3d9087be4919cb90eb69801504114c09923862ff6af"} Nov 01 22:35:03.700716 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:03.700687 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:35:03.713472 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:03.713411 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs" podStartSLOduration=1.713391911 podStartE2EDuration="1.713391911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:03.712863247 +0000 UTC m=+672.117453906" watchObservedRunningTime="2026-11-01 22:35:03.713391911 +0000 UTC m=+672.117982548" Nov 01 22:35:04.695518 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:04.695484 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk" event={"ID":"abea67e1-7f6b-4abd-a1f1-13a2b1613458","Type":"ContainerStarted","Data":"7af8b71d05d028efb8b8aae590fe687c5e12cd5fc70e25162ff94a40d45b95d0"} Nov 01 22:35:04.696043 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:04.695599 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs" Nov 01 22:35:04.729778 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:04.729640 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk" podStartSLOduration=2.277346642 podStartE2EDuration="2.729620092s" totalImagesPullingTime="452.27345ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:04.72739905 +0000 UTC m=+673.131989695" watchObservedRunningTime="2026-11-01 22:35:04.729620092 +0000 UTC m=+673.134210725" Nov 01 22:35:05.565856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:05.565372 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk"] Nov 01 22:35:05.576124 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:05.576091 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk" Nov 01 22:35:05.584454 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:05.584423 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk"] Nov 01 22:35:05.586244 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:05.586216 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-9ad303-serving-cert\"" Nov 01 22:35:05.673523 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:05.673484 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eef0c69-a918-4275-a819-7aed191fd239-metrics-certs\") pod \"maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk\" (UID: \"5eef0c69-a918-4275-a819-7aed191fd239\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk" Nov 01 22:35:05.673523 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:05.673529 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7stt\" (UniqueName: \"kubernetes.io/projected/5eef0c69-a918-4275-a819-7aed191fd239-kube-api-access-k7stt\") pod \"maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk\" (UID: \"5eef0c69-a918-4275-a819-7aed191fd239\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk" Nov 01 22:35:05.673751 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:05.673618 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eef0c69-a918-4275-a819-7aed191fd239-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk\" (UID: \"5eef0c69-a918-4275-a819-7aed191fd239\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk" Nov 01 22:35:05.673751 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:05.673657 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eef0c69-a918-4275-a819-7aed191fd239-maas-api-tls\") pod \"maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk\" (UID: \"5eef0c69-a918-4275-a819-7aed191fd239\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk" Nov 01 22:35:05.709519 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:05.709355 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk" Nov 01 22:35:05.774484 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:05.774442 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eef0c69-a918-4275-a819-7aed191fd239-metrics-certs\") pod \"maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk\" (UID: \"5eef0c69-a918-4275-a819-7aed191fd239\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk" Nov 01 22:35:05.774687 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:05.774622 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7stt\" (UniqueName: \"kubernetes.io/projected/5eef0c69-a918-4275-a819-7aed191fd239-kube-api-access-k7stt\") pod \"maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk\" (UID: \"5eef0c69-a918-4275-a819-7aed191fd239\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk" Nov 01 22:35:05.774754 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:05.774725 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eef0c69-a918-4275-a819-7aed191fd239-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk\" (UID: \"5eef0c69-a918-4275-a819-7aed191fd239\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk" Nov 01 22:35:05.774902 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:05.774877 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eef0c69-a918-4275-a819-7aed191fd239-maas-api-tls\") pod \"maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk\" (UID: \"5eef0c69-a918-4275-a819-7aed191fd239\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk" Nov 01 22:35:05.775338 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:05.775318 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eef0c69-a918-4275-a819-7aed191fd239-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk\" (UID: \"5eef0c69-a918-4275-a819-7aed191fd239\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk" Nov 01 22:35:05.777178 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:05.777150 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eef0c69-a918-4275-a819-7aed191fd239-metrics-certs\") pod \"maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk\" (UID: \"5eef0c69-a918-4275-a819-7aed191fd239\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk" Nov 01 22:35:05.777295 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:05.777275 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eef0c69-a918-4275-a819-7aed191fd239-maas-api-tls\") pod \"maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk\" (UID: \"5eef0c69-a918-4275-a819-7aed191fd239\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk" Nov 01 22:35:05.784202 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:05.784180 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7stt\" (UniqueName: \"kubernetes.io/projected/5eef0c69-a918-4275-a819-7aed191fd239-kube-api-access-k7stt\") pod \"maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk\" (UID: \"5eef0c69-a918-4275-a819-7aed191fd239\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk" Nov 01 22:35:05.895586 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:05.895503 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk" Nov 01 22:35:06.193735 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:06.193695 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk"] Nov 01 22:35:06.196636 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:35:06.196604 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eef0c69_a918_4275_a819_7aed191fd239.slice/crio-bda7c41b973389cce8b88cb7fb814ebee737c8041e0fca52cd4413fa9e5c3817 WatchSource:0}: Error finding container bda7c41b973389cce8b88cb7fb814ebee737c8041e0fca52cd4413fa9e5c3817: Status 404 returned error can't find the container with id bda7c41b973389cce8b88cb7fb814ebee737c8041e0fca52cd4413fa9e5c3817 Nov 01 22:35:06.716034 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:06.715995 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk" event={"ID":"5eef0c69-a918-4275-a819-7aed191fd239","Type":"ContainerStarted","Data":"bda7c41b973389cce8b88cb7fb814ebee737c8041e0fca52cd4413fa9e5c3817"} Nov 01 22:35:07.684153 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:07.684124 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5" Nov 01 22:35:07.727520 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:07.727472 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk" event={"ID":"5eef0c69-a918-4275-a819-7aed191fd239","Type":"ContainerStarted","Data":"5872f2cf363526cf43730a59bc0ffafa4b89858c0e7dd54bd333288f404973a9"} Nov 01 22:35:07.728044 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:07.727754 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk" Nov 01 22:35:07.772004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:07.771939 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk" podStartSLOduration=2.255159888 podStartE2EDuration="2.771920092s" totalImagesPullingTime="516.760204ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:07.771231338 +0000 UTC m=+676.175821983" watchObservedRunningTime="2026-11-01 22:35:07.771920092 +0000 UTC m=+676.176510727" Nov 01 22:35:08.196718 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:08.196681 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:35:08.226075 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:08.222463 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v"] Nov 01 22:35:08.229321 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:08.226682 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v" Nov 01 22:35:08.238822 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:08.238796 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-f338e4-serving-cert\"" Nov 01 22:35:08.280953 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:08.280913 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v"] Nov 01 22:35:08.318247 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:08.318199 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27a3501e-233e-449f-bbfe-8df5ee0731a6-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v\" (UID: \"27a3501e-233e-449f-bbfe-8df5ee0731a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v" Nov 01 22:35:08.318447 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:08.318335 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z2bj\" (UniqueName: \"kubernetes.io/projected/27a3501e-233e-449f-bbfe-8df5ee0731a6-kube-api-access-2z2bj\") pod \"maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v\" (UID: \"27a3501e-233e-449f-bbfe-8df5ee0731a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v" Nov 01 22:35:08.318447 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:08.318419 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27a3501e-233e-449f-bbfe-8df5ee0731a6-metrics-certs\") pod \"maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v\" (UID: \"27a3501e-233e-449f-bbfe-8df5ee0731a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v" Nov 01 22:35:08.318564 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:08.318452 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27a3501e-233e-449f-bbfe-8df5ee0731a6-maas-api-tls\") pod \"maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v\" (UID: \"27a3501e-233e-449f-bbfe-8df5ee0731a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v" Nov 01 22:35:08.419002 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:08.418964 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27a3501e-233e-449f-bbfe-8df5ee0731a6-metrics-certs\") pod \"maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v\" (UID: \"27a3501e-233e-449f-bbfe-8df5ee0731a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v" Nov 01 22:35:08.419189 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:08.419020 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27a3501e-233e-449f-bbfe-8df5ee0731a6-maas-api-tls\") pod \"maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v\" (UID: \"27a3501e-233e-449f-bbfe-8df5ee0731a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v" Nov 01 22:35:08.419189 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:08.419047 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27a3501e-233e-449f-bbfe-8df5ee0731a6-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v\" (UID: \"27a3501e-233e-449f-bbfe-8df5ee0731a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v" Nov 01 22:35:08.419265 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:08.419226 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z2bj\" (UniqueName: \"kubernetes.io/projected/27a3501e-233e-449f-bbfe-8df5ee0731a6-kube-api-access-2z2bj\") pod \"maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v\" (UID: \"27a3501e-233e-449f-bbfe-8df5ee0731a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v" Nov 01 22:35:08.419901 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:08.419877 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27a3501e-233e-449f-bbfe-8df5ee0731a6-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v\" (UID: \"27a3501e-233e-449f-bbfe-8df5ee0731a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v" Nov 01 22:35:08.422332 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:08.422307 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27a3501e-233e-449f-bbfe-8df5ee0731a6-maas-api-tls\") pod \"maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v\" (UID: \"27a3501e-233e-449f-bbfe-8df5ee0731a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v" Nov 01 22:35:08.423334 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:08.423301 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27a3501e-233e-449f-bbfe-8df5ee0731a6-metrics-certs\") pod \"maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v\" (UID: \"27a3501e-233e-449f-bbfe-8df5ee0731a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v" Nov 01 22:35:08.429174 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:08.429147 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z2bj\" (UniqueName: \"kubernetes.io/projected/27a3501e-233e-449f-bbfe-8df5ee0731a6-kube-api-access-2z2bj\") pod \"maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v\" (UID: \"27a3501e-233e-449f-bbfe-8df5ee0731a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v" Nov 01 22:35:08.601905 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:08.598599 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v" Nov 01 22:35:09.054868 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:09.054513 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v"] Nov 01 22:35:09.081152 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:35:09.070112 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a3501e_233e_449f_bbfe_8df5ee0731a6.slice/crio-166cf62d7b379918ad1b194972b5e0eec22173b42f86999d9d49c74e211d08f2 WatchSource:0}: Error finding container 166cf62d7b379918ad1b194972b5e0eec22173b42f86999d9d49c74e211d08f2: Status 404 returned error can't find the container with id 166cf62d7b379918ad1b194972b5e0eec22173b42f86999d9d49c74e211d08f2 Nov 01 22:35:09.778913 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:09.778523 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v" podStartSLOduration=1.373971267 podStartE2EDuration="1.778503697s" totalImagesPullingTime="404.53243ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:09.776587655 +0000 UTC m=+678.181178288" watchObservedRunningTime="2026-11-01 22:35:09.778503697 +0000 UTC m=+678.183094329" Nov 01 22:35:09.793859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:09.787987 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v" Nov 01 22:35:09.793859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:09.788054 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v" event={"ID":"27a3501e-233e-449f-bbfe-8df5ee0731a6","Type":"ContainerStarted","Data":"e90807abd205f6ce8765510b43d155924c4fb10249964f02cc0122beb6a0affc"} Nov 01 22:35:09.793859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:09.788087 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v" event={"ID":"27a3501e-233e-449f-bbfe-8df5ee0731a6","Type":"ContainerStarted","Data":"166cf62d7b379918ad1b194972b5e0eec22173b42f86999d9d49c74e211d08f2"} Nov 01 22:35:11.724091 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:11.724056 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk" Nov 01 22:35:13.703591 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:13.703184 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-2f3705-64d47ccbdf-xgv67" Nov 01 22:35:13.703591 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:13.703277 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2f3705-855984bb47-ztmpt" Nov 01 22:35:13.703591 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:13.703492 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-e5f47e-6b9bd944c5-jbtnp" Nov 01 22:35:13.782524 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:13.782489 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk" Nov 01 22:35:15.713284 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:15.713250 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs" Nov 01 22:35:15.773693 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:15.773638 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v" Nov 01 22:35:15.825687 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:15.825648 5011 generic.go:356] "Generic (PLEG): container finished" podID="b0d043d9-49a8-42fb-872c-cd5479590b1f" containerID="3d39ef4a83cc4951b1891e0c772456f2b9022de0303a5895c61b9e1bb550f43d" exitCode=0 Nov 01 22:35:15.825935 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:15.825721 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" event={"ID":"b0d043d9-49a8-42fb-872c-cd5479590b1f","Type":"ContainerDied","Data":"3d39ef4a83cc4951b1891e0c772456f2b9022de0303a5895c61b9e1bb550f43d"} Nov 01 22:35:15.826260 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:15.826238 5011 scope.go:122] "RemoveContainer" containerID="3d39ef4a83cc4951b1891e0c772456f2b9022de0303a5895c61b9e1bb550f43d" Nov 01 22:35:15.828219 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:15.828182 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:35:15.828342 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:15.828279 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:35:15.828342 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:15.828322 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:35:16.425900 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:16.425860 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:35:16.850786 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:16.850694 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" event={"ID":"b0d043d9-49a8-42fb-872c-cd5479590b1f","Type":"ContainerStarted","Data":"9d66921f3d5de5fe46e049a80484bf43c6a5aa3c1f8802caa53dd027827571db"} Nov 01 22:35:17.022846 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:35:17.022766 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:35:17.834250 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:17.834218 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:35:17.834451 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:17.834337 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 01 22:35:17.834451 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:17.834391 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" podUID="b0d043d9-49a8-42fb-872c-cd5479590b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 01 22:35:18.832020 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:18.831984 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 01 22:35:18.832407 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:18.832038 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" podUID="b0d043d9-49a8-42fb-872c-cd5479590b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 01 22:35:19.832217 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:19.832180 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 01 22:35:19.832654 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:19.832242 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" podUID="b0d043d9-49a8-42fb-872c-cd5479590b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 01 22:35:19.874319 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:19.874286 5011 generic.go:356] "Generic (PLEG): container finished" podID="02d4e19a-0aea-4d86-9876-23181ac12a4a" containerID="a9c4ce39758d1535eabde8e9d66a55bc4feeed6d5022437cdcbf75a485cb5887" exitCode=0 Nov 01 22:35:19.874508 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:19.874369 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" event={"ID":"02d4e19a-0aea-4d86-9876-23181ac12a4a","Type":"ContainerDied","Data":"a9c4ce39758d1535eabde8e9d66a55bc4feeed6d5022437cdcbf75a485cb5887"} Nov 01 22:35:19.874927 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:19.874909 5011 scope.go:122] "RemoveContainer" containerID="a9c4ce39758d1535eabde8e9d66a55bc4feeed6d5022437cdcbf75a485cb5887" Nov 01 22:35:19.876339 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:19.876311 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:35:19.876415 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:19.876392 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:35:19.876457 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:19.876424 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:35:20.847862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:20.846953 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 01 22:35:20.847862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:20.847017 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" podUID="b0d043d9-49a8-42fb-872c-cd5479590b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 01 22:35:20.918857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:20.918604 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" event={"ID":"02d4e19a-0aea-4d86-9876-23181ac12a4a","Type":"ContainerStarted","Data":"6708ab383780e34dbac979eadf2f9ac8fc511ecf01de2e99700f37c7f0d364a2"} Nov 01 22:35:21.579903 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:21.579869 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:35:21.580323 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:21.580296 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 01 22:35:21.580501 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:21.580474 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" podUID="02d4e19a-0aea-4d86-9876-23181ac12a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 01 22:35:21.832238 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:21.832138 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 01 22:35:21.832238 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:21.832216 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" podUID="b0d043d9-49a8-42fb-872c-cd5479590b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 01 22:35:22.580883 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:22.577258 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 01 22:35:22.580883 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:22.577320 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" podUID="02d4e19a-0aea-4d86-9876-23181ac12a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 01 22:35:22.836893 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:22.836810 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:35:22.837799 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:22.837783 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:35:22.838605 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:22.838583 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 22:35:23.108586 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:23.108488 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:35:23.579257 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:23.579220 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 01 22:35:23.579546 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:23.579516 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" podUID="02d4e19a-0aea-4d86-9876-23181ac12a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 01 22:35:24.280861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:24.248556 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t"] Nov 01 22:35:24.327717 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:24.327677 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t"] Nov 01 22:35:24.328016 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:24.327968 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:35:24.338113 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:24.338089 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9ad303\"/\"simulator-w2-9ad303-kserve-self-signed-certs\"" Nov 01 22:35:24.338283 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:24.338131 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9ad303\"/\"default-dockercfg-p5mnk\"" Nov 01 22:35:24.373315 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:24.373006 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9ad303\"/\"kube-root-ca.crt\"" Nov 01 22:35:24.373315 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:24.373173 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9ad303\"/\"openshift-service-ca.crt\"" Nov 01 22:35:24.440873 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:24.440782 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-model-cache\") pod \"simulator-w2-9ad303-kserve-79774d57f9-cs89t\" (UID: \"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0\") " pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:35:24.444606 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:24.444567 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wc2g\" (UniqueName: \"kubernetes.io/projected/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-kube-api-access-2wc2g\") pod \"simulator-w2-9ad303-kserve-79774d57f9-cs89t\" (UID: \"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0\") " pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:35:24.456198 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:24.456162 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-home\") pod \"simulator-w2-9ad303-kserve-79774d57f9-cs89t\" (UID: \"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0\") " pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:35:24.456553 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:24.456532 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-tls-certs\") pod \"simulator-w2-9ad303-kserve-79774d57f9-cs89t\" (UID: \"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0\") " pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:35:24.456688 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:24.456674 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-dshm\") pod \"simulator-w2-9ad303-kserve-79774d57f9-cs89t\" (UID: \"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0\") " pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:35:24.456789 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:24.456777 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-tmp-dir\") pod \"simulator-w2-9ad303-kserve-79774d57f9-cs89t\" (UID: \"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0\") " pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:35:24.561919 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:24.561471 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-tls-certs\") pod \"simulator-w2-9ad303-kserve-79774d57f9-cs89t\" (UID: \"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0\") " pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:35:24.561919 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:24.561533 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-dshm\") pod \"simulator-w2-9ad303-kserve-79774d57f9-cs89t\" (UID: \"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0\") " pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:35:24.561919 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:24.561570 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-tmp-dir\") pod \"simulator-w2-9ad303-kserve-79774d57f9-cs89t\" (UID: \"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0\") " pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:35:24.561919 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:24.561619 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-model-cache\") pod \"simulator-w2-9ad303-kserve-79774d57f9-cs89t\" (UID: \"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0\") " pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:35:24.561919 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:24.561685 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wc2g\" (UniqueName: \"kubernetes.io/projected/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-kube-api-access-2wc2g\") pod \"simulator-w2-9ad303-kserve-79774d57f9-cs89t\" (UID: \"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0\") " pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:35:24.561919 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:24.561729 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-home\") pod \"simulator-w2-9ad303-kserve-79774d57f9-cs89t\" (UID: \"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0\") " pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:35:24.562395 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:24.562236 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-home\") pod \"simulator-w2-9ad303-kserve-79774d57f9-cs89t\" (UID: \"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0\") " pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:35:24.572964 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:24.562903 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-tmp-dir\") pod \"simulator-w2-9ad303-kserve-79774d57f9-cs89t\" (UID: \"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0\") " pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:35:24.573486 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:24.573459 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-model-cache\") pod \"simulator-w2-9ad303-kserve-79774d57f9-cs89t\" (UID: \"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0\") " pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:35:24.595009 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:24.594975 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 01 22:35:24.595259 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:24.595232 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" podUID="02d4e19a-0aea-4d86-9876-23181ac12a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 01 22:35:24.597551 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:24.597525 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-dshm\") pod \"simulator-w2-9ad303-kserve-79774d57f9-cs89t\" (UID: \"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0\") " pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:35:24.597740 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:24.597643 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-tls-certs\") pod \"simulator-w2-9ad303-kserve-79774d57f9-cs89t\" (UID: \"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0\") " pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:35:24.600586 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:24.600561 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wc2g\" (UniqueName: \"kubernetes.io/projected/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-kube-api-access-2wc2g\") pod \"simulator-w2-9ad303-kserve-79774d57f9-cs89t\" (UID: \"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0\") " pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:35:24.729786 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:24.729745 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:35:25.132435 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:25.132363 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t"] Nov 01 22:35:25.135103 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:35:25.135060 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a134ecf_28e8_41a3_ac49_e56fa1b93dd0.slice/crio-f7e7fd4072e7243a7be4de151a978f773f6c302783a4d67c86bb21dfd8d38c48 WatchSource:0}: Error finding container f7e7fd4072e7243a7be4de151a978f773f6c302783a4d67c86bb21dfd8d38c48: Status 404 returned error can't find the container with id f7e7fd4072e7243a7be4de151a978f773f6c302783a4d67c86bb21dfd8d38c48 Nov 01 22:35:25.580692 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:25.580249 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 01 22:35:25.580692 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:25.580311 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" podUID="02d4e19a-0aea-4d86-9876-23181ac12a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 01 22:35:25.606976 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:25.606104 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:35:26.125767 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:26.125729 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" event={"ID":"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0","Type":"ContainerStarted","Data":"56acf032d1ff9cfa5283c04799f6258894976d090fd4e2761aaf844ae04f8c59"} Nov 01 22:35:26.125767 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:26.125772 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" event={"ID":"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0","Type":"ContainerStarted","Data":"f7e7fd4072e7243a7be4de151a978f773f6c302783a4d67c86bb21dfd8d38c48"} Nov 01 22:35:26.163858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:26.148522 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podStartSLOduration=2.148501213 podStartE2EDuration="2.148501213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:26.145240441 +0000 UTC m=+694.549831075" watchObservedRunningTime="2026-11-01 22:35:26.148501213 +0000 UTC m=+694.553091843" Nov 01 22:35:26.581384 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:26.581354 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:35:26.582451 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:26.582420 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:35:26.730329 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:26.730295 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:35:31.159226 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:31.159182 5011 generic.go:356] "Generic (PLEG): container finished" podID="b5730c77-aecb-441e-82f5-9d34e5e3b8d5" containerID="444caa99c6b01836eaa1d41f577683c7cf47b49f05483f5bc886609c980aab3c" exitCode=0 Nov 01 22:35:31.159724 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:31.159253 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" event={"ID":"b5730c77-aecb-441e-82f5-9d34e5e3b8d5","Type":"ContainerDied","Data":"444caa99c6b01836eaa1d41f577683c7cf47b49f05483f5bc886609c980aab3c"} Nov 01 22:35:31.159724 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:31.159654 5011 scope.go:122] "RemoveContainer" containerID="444caa99c6b01836eaa1d41f577683c7cf47b49f05483f5bc886609c980aab3c" Nov 01 22:35:31.161937 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:31.161895 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:35:31.162083 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:31.162062 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:35:31.162301 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:31.162283 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:35:32.195131 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:32.195087 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" event={"ID":"b5730c77-aecb-441e-82f5-9d34e5e3b8d5","Type":"ContainerStarted","Data":"c377a821828cc93fe4c52895c50a58c05f2e499672010ab8b25e5285758194e0"} Nov 01 22:35:33.085208 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:33.085178 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:35:33.085420 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:33.085397 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 01 22:35:33.085479 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:33.085456 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" podUID="b5730c77-aecb-441e-82f5-9d34e5e3b8d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 01 22:35:34.084448 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:34.084409 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 01 22:35:34.084875 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:34.084463 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" podUID="b5730c77-aecb-441e-82f5-9d34e5e3b8d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 01 22:35:34.730712 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:34.730675 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:35:34.767865 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:34.767798 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:35:35.085074 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:35.084977 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 01 22:35:35.085074 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:35.085041 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" podUID="b5730c77-aecb-441e-82f5-9d34e5e3b8d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 01 22:35:35.227589 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:35.227548 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:35:35.331616 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:35.331580 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:35:35.404311 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:35.404219 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr"] Nov 01 22:35:35.408725 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:35.408665 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:35:35.420306 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:35.420276 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e5f47e\"/\"default-dockercfg-zngpk\"" Nov 01 22:35:35.420479 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:35.420392 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e5f47e\"/\"kube-root-ca.crt\"" Nov 01 22:35:35.420542 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:35.420496 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e5f47e\"/\"openshift-service-ca.crt\"" Nov 01 22:35:35.420657 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:35.420639 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e5f47e\"/\"simulator-w0-e5f47e-kserve-self-signed-certs\"" Nov 01 22:35:35.420882 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:35.420864 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr"] Nov 01 22:35:35.509572 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:35.509534 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74441b26-2636-41c2-84e4-6a6be32d2793-home\") pod \"simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr\" (UID: \"74441b26-2636-41c2-84e4-6a6be32d2793\") " pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:35:35.509572 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:35.509577 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zb8n\" (UniqueName: \"kubernetes.io/projected/74441b26-2636-41c2-84e4-6a6be32d2793-kube-api-access-5zb8n\") pod \"simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr\" (UID: \"74441b26-2636-41c2-84e4-6a6be32d2793\") " pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:35:35.509854 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:35.509733 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74441b26-2636-41c2-84e4-6a6be32d2793-tmp-dir\") pod \"simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr\" (UID: \"74441b26-2636-41c2-84e4-6a6be32d2793\") " pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:35:35.509854 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:35.509785 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74441b26-2636-41c2-84e4-6a6be32d2793-model-cache\") pod \"simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr\" (UID: \"74441b26-2636-41c2-84e4-6a6be32d2793\") " pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:35:35.509929 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:35.509894 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74441b26-2636-41c2-84e4-6a6be32d2793-tls-certs\") pod \"simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr\" (UID: \"74441b26-2636-41c2-84e4-6a6be32d2793\") " pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:35:35.509969 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:35.509937 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74441b26-2636-41c2-84e4-6a6be32d2793-dshm\") pod \"simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr\" (UID: \"74441b26-2636-41c2-84e4-6a6be32d2793\") " pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:35:35.614856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:35.613698 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74441b26-2636-41c2-84e4-6a6be32d2793-model-cache\") pod \"simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr\" (UID: \"74441b26-2636-41c2-84e4-6a6be32d2793\") " pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:35:35.614856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:35.614289 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74441b26-2636-41c2-84e4-6a6be32d2793-model-cache\") pod \"simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr\" (UID: \"74441b26-2636-41c2-84e4-6a6be32d2793\") " pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:35:35.614856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:35.614338 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74441b26-2636-41c2-84e4-6a6be32d2793-tls-certs\") pod \"simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr\" (UID: \"74441b26-2636-41c2-84e4-6a6be32d2793\") " pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:35:35.615170 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:35.614870 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74441b26-2636-41c2-84e4-6a6be32d2793-dshm\") pod \"simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr\" (UID: \"74441b26-2636-41c2-84e4-6a6be32d2793\") " pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:35:35.615170 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:35.614958 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74441b26-2636-41c2-84e4-6a6be32d2793-home\") pod \"simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr\" (UID: \"74441b26-2636-41c2-84e4-6a6be32d2793\") " pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:35:35.615170 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:35.614987 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zb8n\" (UniqueName: \"kubernetes.io/projected/74441b26-2636-41c2-84e4-6a6be32d2793-kube-api-access-5zb8n\") pod \"simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr\" (UID: \"74441b26-2636-41c2-84e4-6a6be32d2793\") " pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:35:35.615170 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:35.615106 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74441b26-2636-41c2-84e4-6a6be32d2793-tmp-dir\") pod \"simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr\" (UID: \"74441b26-2636-41c2-84e4-6a6be32d2793\") " pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:35:35.616165 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:35.615450 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74441b26-2636-41c2-84e4-6a6be32d2793-tmp-dir\") pod \"simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr\" (UID: \"74441b26-2636-41c2-84e4-6a6be32d2793\") " pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:35:35.624859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:35.617536 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74441b26-2636-41c2-84e4-6a6be32d2793-home\") pod \"simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr\" (UID: \"74441b26-2636-41c2-84e4-6a6be32d2793\") " pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:35:35.624859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:35.622852 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74441b26-2636-41c2-84e4-6a6be32d2793-dshm\") pod \"simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr\" (UID: \"74441b26-2636-41c2-84e4-6a6be32d2793\") " pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:35:35.627523 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:35.627422 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74441b26-2636-41c2-84e4-6a6be32d2793-tls-certs\") pod \"simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr\" (UID: \"74441b26-2636-41c2-84e4-6a6be32d2793\") " pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:35:35.641951 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:35.641922 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zb8n\" (UniqueName: \"kubernetes.io/projected/74441b26-2636-41c2-84e4-6a6be32d2793-kube-api-access-5zb8n\") pod \"simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr\" (UID: \"74441b26-2636-41c2-84e4-6a6be32d2793\") " pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:35:35.726692 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:35.726653 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:35:36.085043 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:36.084953 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 01 22:35:36.085043 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:36.085022 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" podUID="b5730c77-aecb-441e-82f5-9d34e5e3b8d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 01 22:35:36.115051 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:36.114010 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr"] Nov 01 22:35:36.116641 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:35:36.116599 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74441b26_2636_41c2_84e4_6a6be32d2793.slice/crio-a75cec771050da40bb02b0229efb9f53d2b0da64502c1d14e27cb20b6b8f77e5 WatchSource:0}: Error finding container a75cec771050da40bb02b0229efb9f53d2b0da64502c1d14e27cb20b6b8f77e5: Status 404 returned error can't find the container with id a75cec771050da40bb02b0229efb9f53d2b0da64502c1d14e27cb20b6b8f77e5 Nov 01 22:35:36.274902 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:36.268247 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" event={"ID":"74441b26-2636-41c2-84e4-6a6be32d2793","Type":"ContainerStarted","Data":"a75cec771050da40bb02b0229efb9f53d2b0da64502c1d14e27cb20b6b8f77e5"} Nov 01 22:35:37.084799 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:37.084762 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 01 22:35:37.085014 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:37.084844 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" podUID="b5730c77-aecb-441e-82f5-9d34e5e3b8d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 01 22:35:37.275511 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:37.275469 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" event={"ID":"74441b26-2636-41c2-84e4-6a6be32d2793","Type":"ContainerStarted","Data":"1035e6b86781a2fb350ce93fda836ceb88d7316d442d678b9dabbf3e6ea56764"} Nov 01 22:35:37.294594 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:37.294532 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podStartSLOduration=2.294517328 podStartE2EDuration="2.294517328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:37.294023712 +0000 UTC m=+705.698614339" watchObservedRunningTime="2026-11-01 22:35:37.294517328 +0000 UTC m=+705.699107983" Nov 01 22:35:37.727281 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:37.727242 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:35:38.089539 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:38.089462 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:35:38.089764 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:38.089746 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:35:38.090533 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:38.090514 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:35:44.646858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:44.644986 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:35:45.731853 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:45.731186 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:35:45.756218 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:45.756188 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:35:46.344702 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:46.344671 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:35:46.960195 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:35:46.960153 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:35:48.314362 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:48.314326 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz"] Nov 01 22:35:48.318509 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:48.318486 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:35:48.327673 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:48.327648 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-927b68\"/\"kube-root-ca.crt\"" Nov 01 22:35:48.328525 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:48.328497 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-927b68\"/\"openshift-service-ca.crt\"" Nov 01 22:35:48.328669 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:48.328569 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-927b68\"/\"premium-w6-927b68-kserve-self-signed-certs\"" Nov 01 22:35:48.328732 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:48.328664 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-927b68\"/\"default-dockercfg-s59jf\"" Nov 01 22:35:48.330619 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:48.330592 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz"] Nov 01 22:35:48.400295 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:48.400258 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3508619c-ddff-4d19-8e29-a28cb562de3b-dshm\") pod \"premium-w6-927b68-kserve-7779999776-589fz\" (UID: \"3508619c-ddff-4d19-8e29-a28cb562de3b\") " pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:35:48.400490 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:48.400303 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3508619c-ddff-4d19-8e29-a28cb562de3b-tls-certs\") pod \"premium-w6-927b68-kserve-7779999776-589fz\" (UID: \"3508619c-ddff-4d19-8e29-a28cb562de3b\") " pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:35:48.400490 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:48.400403 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3508619c-ddff-4d19-8e29-a28cb562de3b-model-cache\") pod \"premium-w6-927b68-kserve-7779999776-589fz\" (UID: \"3508619c-ddff-4d19-8e29-a28cb562de3b\") " pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:35:48.400588 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:48.400565 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3508619c-ddff-4d19-8e29-a28cb562de3b-tmp-dir\") pod \"premium-w6-927b68-kserve-7779999776-589fz\" (UID: \"3508619c-ddff-4d19-8e29-a28cb562de3b\") " pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:35:48.400637 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:48.400624 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtnz5\" (UniqueName: \"kubernetes.io/projected/3508619c-ddff-4d19-8e29-a28cb562de3b-kube-api-access-mtnz5\") pod \"premium-w6-927b68-kserve-7779999776-589fz\" (UID: \"3508619c-ddff-4d19-8e29-a28cb562de3b\") " pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:35:48.400724 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:48.400707 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3508619c-ddff-4d19-8e29-a28cb562de3b-home\") pod \"premium-w6-927b68-kserve-7779999776-589fz\" (UID: \"3508619c-ddff-4d19-8e29-a28cb562de3b\") " pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:35:48.501785 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:48.501701 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3508619c-ddff-4d19-8e29-a28cb562de3b-model-cache\") pod \"premium-w6-927b68-kserve-7779999776-589fz\" (UID: \"3508619c-ddff-4d19-8e29-a28cb562de3b\") " pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:35:48.502004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:48.501824 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3508619c-ddff-4d19-8e29-a28cb562de3b-tmp-dir\") pod \"premium-w6-927b68-kserve-7779999776-589fz\" (UID: \"3508619c-ddff-4d19-8e29-a28cb562de3b\") " pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:35:48.502004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:48.501908 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtnz5\" (UniqueName: \"kubernetes.io/projected/3508619c-ddff-4d19-8e29-a28cb562de3b-kube-api-access-mtnz5\") pod \"premium-w6-927b68-kserve-7779999776-589fz\" (UID: \"3508619c-ddff-4d19-8e29-a28cb562de3b\") " pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:35:48.502004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:48.501944 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3508619c-ddff-4d19-8e29-a28cb562de3b-home\") pod \"premium-w6-927b68-kserve-7779999776-589fz\" (UID: \"3508619c-ddff-4d19-8e29-a28cb562de3b\") " pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:35:48.502176 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:48.502032 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3508619c-ddff-4d19-8e29-a28cb562de3b-dshm\") pod \"premium-w6-927b68-kserve-7779999776-589fz\" (UID: \"3508619c-ddff-4d19-8e29-a28cb562de3b\") " pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:35:48.502176 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:48.502056 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3508619c-ddff-4d19-8e29-a28cb562de3b-tls-certs\") pod \"premium-w6-927b68-kserve-7779999776-589fz\" (UID: \"3508619c-ddff-4d19-8e29-a28cb562de3b\") " pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:35:48.502324 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:48.502298 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3508619c-ddff-4d19-8e29-a28cb562de3b-model-cache\") pod \"premium-w6-927b68-kserve-7779999776-589fz\" (UID: \"3508619c-ddff-4d19-8e29-a28cb562de3b\") " pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:35:48.502672 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:48.502641 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3508619c-ddff-4d19-8e29-a28cb562de3b-tmp-dir\") pod \"premium-w6-927b68-kserve-7779999776-589fz\" (UID: \"3508619c-ddff-4d19-8e29-a28cb562de3b\") " pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:35:48.502969 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:48.502950 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3508619c-ddff-4d19-8e29-a28cb562de3b-home\") pod \"premium-w6-927b68-kserve-7779999776-589fz\" (UID: \"3508619c-ddff-4d19-8e29-a28cb562de3b\") " pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:35:48.504803 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:48.504776 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3508619c-ddff-4d19-8e29-a28cb562de3b-dshm\") pod \"premium-w6-927b68-kserve-7779999776-589fz\" (UID: \"3508619c-ddff-4d19-8e29-a28cb562de3b\") " pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:35:48.505015 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:48.504997 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3508619c-ddff-4d19-8e29-a28cb562de3b-tls-certs\") pod \"premium-w6-927b68-kserve-7779999776-589fz\" (UID: \"3508619c-ddff-4d19-8e29-a28cb562de3b\") " pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:35:48.514353 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:48.514331 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtnz5\" (UniqueName: \"kubernetes.io/projected/3508619c-ddff-4d19-8e29-a28cb562de3b-kube-api-access-mtnz5\") pod \"premium-w6-927b68-kserve-7779999776-589fz\" (UID: \"3508619c-ddff-4d19-8e29-a28cb562de3b\") " pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:35:48.632727 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:48.632629 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:35:48.974362 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:48.974335 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz"] Nov 01 22:35:48.976912 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:35:48.976883 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3508619c_ddff_4d19_8e29_a28cb562de3b.slice/crio-846b2260ac315d712b5c14b030c3be105c3cd76f53a78f998975b22881609052 WatchSource:0}: Error finding container 846b2260ac315d712b5c14b030c3be105c3cd76f53a78f998975b22881609052: Status 404 returned error can't find the container with id 846b2260ac315d712b5c14b030c3be105c3cd76f53a78f998975b22881609052 Nov 01 22:35:49.351284 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:49.351184 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" event={"ID":"3508619c-ddff-4d19-8e29-a28cb562de3b","Type":"ContainerStarted","Data":"69cff53ae35b42778eb93834c7ef391af268cc2bfcf4141b69cb3791576988f8"} Nov 01 22:35:49.351284 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:49.351236 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" event={"ID":"3508619c-ddff-4d19-8e29-a28cb562de3b","Type":"ContainerStarted","Data":"846b2260ac315d712b5c14b030c3be105c3cd76f53a78f998975b22881609052"} Nov 01 22:35:49.369400 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:49.369342 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podStartSLOduration=1.369327958 podStartE2EDuration="1.369327958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:49.368606117 +0000 UTC m=+717.773196754" watchObservedRunningTime="2026-11-01 22:35:49.369327958 +0000 UTC m=+717.773918646" Nov 01 22:35:49.633066 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:49.632979 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:35:56.212732 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:56.212693 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:35:56.525849 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:56.525742 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw"] Nov 01 22:35:56.531476 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:56.531446 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:35:56.540313 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:56.539730 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-f338e4\"/\"kube-root-ca.crt\"" Nov 01 22:35:56.540399 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:56.540341 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f338e4\"/\"premium-w1-f338e4-kserve-self-signed-certs\"" Nov 01 22:35:56.540629 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:56.540466 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f338e4\"/\"default-dockercfg-td692\"" Nov 01 22:35:56.540629 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:56.540537 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-f338e4\"/\"openshift-service-ca.crt\"" Nov 01 22:35:56.544661 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:56.544634 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw"] Nov 01 22:35:56.603479 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:56.603441 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-model-cache\") pod \"premium-w1-f338e4-kserve-8ff4979d6-mhssw\" (UID: \"8b3ef57b-dc90-49b0-98c6-85eacd323a9b\") " pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:35:56.603479 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:56.603485 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-tls-certs\") pod \"premium-w1-f338e4-kserve-8ff4979d6-mhssw\" (UID: \"8b3ef57b-dc90-49b0-98c6-85eacd323a9b\") " pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:35:56.603730 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:56.603516 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-dshm\") pod \"premium-w1-f338e4-kserve-8ff4979d6-mhssw\" (UID: \"8b3ef57b-dc90-49b0-98c6-85eacd323a9b\") " pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:35:56.603730 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:56.603709 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-tmp-dir\") pod \"premium-w1-f338e4-kserve-8ff4979d6-mhssw\" (UID: \"8b3ef57b-dc90-49b0-98c6-85eacd323a9b\") " pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:35:56.603895 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:56.603878 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k7fs\" (UniqueName: \"kubernetes.io/projected/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-kube-api-access-7k7fs\") pod \"premium-w1-f338e4-kserve-8ff4979d6-mhssw\" (UID: \"8b3ef57b-dc90-49b0-98c6-85eacd323a9b\") " pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:35:56.603954 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:56.603911 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-home\") pod \"premium-w1-f338e4-kserve-8ff4979d6-mhssw\" (UID: \"8b3ef57b-dc90-49b0-98c6-85eacd323a9b\") " pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:35:56.705164 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:56.705126 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k7fs\" (UniqueName: \"kubernetes.io/projected/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-kube-api-access-7k7fs\") pod \"premium-w1-f338e4-kserve-8ff4979d6-mhssw\" (UID: \"8b3ef57b-dc90-49b0-98c6-85eacd323a9b\") " pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:35:56.705164 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:56.705166 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-home\") pod \"premium-w1-f338e4-kserve-8ff4979d6-mhssw\" (UID: \"8b3ef57b-dc90-49b0-98c6-85eacd323a9b\") " pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:35:56.705400 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:56.705197 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-model-cache\") pod \"premium-w1-f338e4-kserve-8ff4979d6-mhssw\" (UID: \"8b3ef57b-dc90-49b0-98c6-85eacd323a9b\") " pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:35:56.705400 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:56.705225 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-tls-certs\") pod \"premium-w1-f338e4-kserve-8ff4979d6-mhssw\" (UID: \"8b3ef57b-dc90-49b0-98c6-85eacd323a9b\") " pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:35:56.705400 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:56.705256 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-dshm\") pod \"premium-w1-f338e4-kserve-8ff4979d6-mhssw\" (UID: \"8b3ef57b-dc90-49b0-98c6-85eacd323a9b\") " pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:35:56.705400 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:56.705321 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-tmp-dir\") pod \"premium-w1-f338e4-kserve-8ff4979d6-mhssw\" (UID: \"8b3ef57b-dc90-49b0-98c6-85eacd323a9b\") " pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:35:56.705811 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:56.705785 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-home\") pod \"premium-w1-f338e4-kserve-8ff4979d6-mhssw\" (UID: \"8b3ef57b-dc90-49b0-98c6-85eacd323a9b\") " pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:35:56.705934 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:56.705822 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-tmp-dir\") pod \"premium-w1-f338e4-kserve-8ff4979d6-mhssw\" (UID: \"8b3ef57b-dc90-49b0-98c6-85eacd323a9b\") " pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:35:56.706288 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:56.706261 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-model-cache\") pod \"premium-w1-f338e4-kserve-8ff4979d6-mhssw\" (UID: \"8b3ef57b-dc90-49b0-98c6-85eacd323a9b\") " pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:35:56.707962 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:56.707937 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-dshm\") pod \"premium-w1-f338e4-kserve-8ff4979d6-mhssw\" (UID: \"8b3ef57b-dc90-49b0-98c6-85eacd323a9b\") " pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:35:56.708337 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:56.708317 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-tls-certs\") pod \"premium-w1-f338e4-kserve-8ff4979d6-mhssw\" (UID: \"8b3ef57b-dc90-49b0-98c6-85eacd323a9b\") " pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:35:56.714291 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:56.714269 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k7fs\" (UniqueName: \"kubernetes.io/projected/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-kube-api-access-7k7fs\") pod \"premium-w1-f338e4-kserve-8ff4979d6-mhssw\" (UID: \"8b3ef57b-dc90-49b0-98c6-85eacd323a9b\") " pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:35:56.847681 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:56.847588 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:35:57.151877 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:57.151846 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw"] Nov 01 22:35:57.153757 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:35:57.153721 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b3ef57b_dc90_49b0_98c6_85eacd323a9b.slice/crio-c0b08c4a87476e2085edfbb51d143b39f61e12107ee6555772457a08dcda752e WatchSource:0}: Error finding container c0b08c4a87476e2085edfbb51d143b39f61e12107ee6555772457a08dcda752e: Status 404 returned error can't find the container with id c0b08c4a87476e2085edfbb51d143b39f61e12107ee6555772457a08dcda752e Nov 01 22:35:57.402578 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:57.402472 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" event={"ID":"8b3ef57b-dc90-49b0-98c6-85eacd323a9b","Type":"ContainerStarted","Data":"9c6d68132a835dacc44926d95316b3d8c134719b0759ff7299de43db84797e9a"} Nov 01 22:35:57.402578 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:57.402520 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" event={"ID":"8b3ef57b-dc90-49b0-98c6-85eacd323a9b","Type":"ContainerStarted","Data":"c0b08c4a87476e2085edfbb51d143b39f61e12107ee6555772457a08dcda752e"} Nov 01 22:35:57.440876 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:57.440799 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podStartSLOduration=1.440782127 podStartE2EDuration="1.440782127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:57.439891196 +0000 UTC m=+725.844481829" watchObservedRunningTime="2026-11-01 22:35:57.440782127 +0000 UTC m=+725.845372760" Nov 01 22:35:57.848739 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:57.848698 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:35:58.633462 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:58.633434 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:35:58.646328 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:58.646302 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:35:59.425015 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:35:59.424985 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:36:06.848575 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:06.848468 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:36:06.862355 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:06.862324 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:36:07.479909 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:07.479869 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:36:16.959356 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:36:16.959315 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:36:20.505697 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:20.505662 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:36:23.624705 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:23.624666 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9"] Nov 01 22:36:23.630453 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:23.630431 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:23.640157 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:23.640139 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-9ad303-openshift-default-dockercfg-cbwmc\"" Nov 01 22:36:23.660046 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:23.660011 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-workload-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:23.660222 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:23.660105 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed165c04-d90b-45c4-8022-670fb4f2b61c-istiod-ca-cert\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:23.660222 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:23.660154 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-workload-certs\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:23.660222 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:23.660200 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-credential-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:23.660406 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:23.660227 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7gkv\" (UniqueName: \"kubernetes.io/projected/ed165c04-d90b-45c4-8022-670fb4f2b61c-kube-api-access-w7gkv\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:23.660406 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:23.660267 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed165c04-d90b-45c4-8022-670fb4f2b61c-istio-token\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:23.660406 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:23.660301 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-istio-envoy\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:23.660406 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:23.660329 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed165c04-d90b-45c4-8022-670fb4f2b61c-istio-podinfo\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:23.660406 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:23.660365 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-istio-data\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:23.660597 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:23.660495 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9"] Nov 01 22:36:23.761025 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:23.760992 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-workload-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:23.761208 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:23.761059 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed165c04-d90b-45c4-8022-670fb4f2b61c-istiod-ca-cert\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:23.761208 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:23.761095 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-workload-certs\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:23.761208 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:23.761134 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-credential-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:23.761208 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:23.761153 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7gkv\" (UniqueName: \"kubernetes.io/projected/ed165c04-d90b-45c4-8022-670fb4f2b61c-kube-api-access-w7gkv\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:23.761208 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:23.761195 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed165c04-d90b-45c4-8022-670fb4f2b61c-istio-token\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:23.761529 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:23.761230 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-istio-envoy\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:23.761529 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:23.761256 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed165c04-d90b-45c4-8022-670fb4f2b61c-istio-podinfo\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:23.761529 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:23.761290 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-istio-data\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:23.761529 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:23.761489 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-workload-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:23.761781 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:23.761549 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-credential-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:23.761781 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:23.761627 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-workload-certs\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:23.761781 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:23.761650 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-istio-data\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:23.761926 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:23.761904 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed165c04-d90b-45c4-8022-670fb4f2b61c-istiod-ca-cert\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:23.763986 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:23.763961 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-istio-envoy\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:23.764097 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:23.763984 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed165c04-d90b-45c4-8022-670fb4f2b61c-istio-podinfo\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:23.782147 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:23.782122 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7gkv\" (UniqueName: \"kubernetes.io/projected/ed165c04-d90b-45c4-8022-670fb4f2b61c-kube-api-access-w7gkv\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:23.782356 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:23.782335 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed165c04-d90b-45c4-8022-670fb4f2b61c-istio-token\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:23.947432 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:23.947392 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:24.241868 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:24.241817 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9"] Nov 01 22:36:24.272583 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:24.272521 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:36:24.272795 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:24.272624 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:36:24.272795 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:24.272674 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:36:24.597712 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:24.597593 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" event={"ID":"ed165c04-d90b-45c4-8022-670fb4f2b61c","Type":"ContainerStarted","Data":"a8b86cb243231103f2239e4ea6e8444455c12504f542687571e3b6a6802b64e4"} Nov 01 22:36:24.597712 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:24.597699 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" event={"ID":"ed165c04-d90b-45c4-8022-670fb4f2b61c","Type":"ContainerStarted","Data":"616c565465bded9588129389444cf753bcbf1ad18d5a76dad93b13221bc99c63"} Nov 01 22:36:24.642125 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:24.642074 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" podStartSLOduration=1.642060452 podStartE2EDuration="1.642060452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:24.641862567 +0000 UTC m=+753.046453271" watchObservedRunningTime="2026-11-01 22:36:24.642060452 +0000 UTC m=+753.046651084" Nov 01 22:36:24.955860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:24.948947 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:25.966466 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:25.966002 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 01 22:36:25.966466 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:25.966087 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" podUID="ed165c04-d90b-45c4-8022-670fb4f2b61c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 01 22:36:26.950858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:26.948244 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 01 22:36:26.950858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:26.948325 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" podUID="ed165c04-d90b-45c4-8022-670fb4f2b61c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 01 22:36:27.959860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:27.953296 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 01 22:36:27.959860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:27.953357 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" podUID="ed165c04-d90b-45c4-8022-670fb4f2b61c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 01 22:36:28.952858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:28.949209 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 01 22:36:28.952858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:28.949272 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" podUID="ed165c04-d90b-45c4-8022-670fb4f2b61c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 01 22:36:29.960321 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:29.960279 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 01 22:36:29.961006 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:29.960972 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" podUID="ed165c04-d90b-45c4-8022-670fb4f2b61c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 01 22:36:31.025949 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:31.025899 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:31.026599 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:31.026565 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:31.027629 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:31.027606 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 22:36:38.083496 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.083457 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9"] Nov 01 22:36:38.087467 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.087446 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:38.096803 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.096780 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-927b68-openshift-default-dockercfg-rfvvh\"" Nov 01 22:36:38.112388 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.112350 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9"] Nov 01 22:36:38.185021 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.184985 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-istio-envoy\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:38.185021 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.185029 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzxcr\" (UniqueName: \"kubernetes.io/projected/0ee11042-8133-4e65-950e-224a5b4ca910-kube-api-access-dzxcr\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:38.185258 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.185046 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-workload-certs\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:38.185258 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.185114 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ee11042-8133-4e65-950e-224a5b4ca910-istio-podinfo\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:38.185258 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.185245 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-credential-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:38.185372 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.185290 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ee11042-8133-4e65-950e-224a5b4ca910-istio-token\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:38.185372 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.185324 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ee11042-8133-4e65-950e-224a5b4ca910-istiod-ca-cert\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:38.185456 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.185395 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-workload-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:38.185505 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.185490 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-istio-data\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:38.286893 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.286820 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-istio-envoy\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:38.287079 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.286904 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzxcr\" (UniqueName: \"kubernetes.io/projected/0ee11042-8133-4e65-950e-224a5b4ca910-kube-api-access-dzxcr\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:38.287079 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.286931 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-workload-certs\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:38.287079 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.286959 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ee11042-8133-4e65-950e-224a5b4ca910-istio-podinfo\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:38.287079 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.286999 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-credential-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:38.287079 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.287060 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ee11042-8133-4e65-950e-224a5b4ca910-istio-token\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:38.287341 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.287112 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ee11042-8133-4e65-950e-224a5b4ca910-istiod-ca-cert\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:38.287341 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.287165 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-workload-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:38.287341 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.287252 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-istio-data\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:38.287501 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.287392 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-workload-certs\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:38.287501 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.287489 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-credential-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:38.287729 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.287707 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-workload-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:38.287880 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.287856 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ee11042-8133-4e65-950e-224a5b4ca910-istiod-ca-cert\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:38.288002 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.287857 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-istio-data\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:38.289345 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.289327 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-istio-envoy\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:38.290988 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.290969 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ee11042-8133-4e65-950e-224a5b4ca910-istio-podinfo\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:38.297860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.297809 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ee11042-8133-4e65-950e-224a5b4ca910-istio-token\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:38.297974 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.297874 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzxcr\" (UniqueName: \"kubernetes.io/projected/0ee11042-8133-4e65-950e-224a5b4ca910-kube-api-access-dzxcr\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:38.422848 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.422801 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:38.711045 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.711014 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9"] Nov 01 22:36:38.712650 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:36:38.712618 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee11042_8133_4e65_950e_224a5b4ca910.slice/crio-367f6400b09671c3aaf50220bdd93a100d2b64d2a0de848ff6d05d48a30f9d51 WatchSource:0}: Error finding container 367f6400b09671c3aaf50220bdd93a100d2b64d2a0de848ff6d05d48a30f9d51: Status 404 returned error can't find the container with id 367f6400b09671c3aaf50220bdd93a100d2b64d2a0de848ff6d05d48a30f9d51 Nov 01 22:36:38.715405 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.715368 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:36:38.715485 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.715439 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:36:38.715485 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:38.715469 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:36:39.688392 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:39.688355 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" event={"ID":"0ee11042-8133-4e65-950e-224a5b4ca910","Type":"ContainerStarted","Data":"8b3baaf41808f3b42c8e358849ded2c8903da9824640aeaedcf5d9dfe5037a1f"} Nov 01 22:36:39.688796 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:39.688409 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" event={"ID":"0ee11042-8133-4e65-950e-224a5b4ca910","Type":"ContainerStarted","Data":"367f6400b09671c3aaf50220bdd93a100d2b64d2a0de848ff6d05d48a30f9d51"} Nov 01 22:36:39.731205 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:39.731154 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" podStartSLOduration=1.7311372170000001 podStartE2EDuration="1.731137217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:39.729286727 +0000 UTC m=+768.133877396" watchObservedRunningTime="2026-11-01 22:36:39.731137217 +0000 UTC m=+768.135727848" Nov 01 22:36:40.423387 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:40.423350 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:40.425639 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:40.425360 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 01 22:36:40.425639 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:40.425427 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" podUID="0ee11042-8133-4e65-950e-224a5b4ca910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 01 22:36:41.424061 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:41.424020 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 01 22:36:41.424451 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:41.424098 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" podUID="0ee11042-8133-4e65-950e-224a5b4ca910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 01 22:36:42.424033 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:42.423999 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 01 22:36:42.424218 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:42.424064 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" podUID="0ee11042-8133-4e65-950e-224a5b4ca910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 01 22:36:42.908008 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:42.907926 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:36:43.429858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:43.426769 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 01 22:36:43.429858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:43.426871 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" podUID="0ee11042-8133-4e65-950e-224a5b4ca910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 01 22:36:44.423601 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:44.423564 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 01 22:36:44.423786 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:44.423636 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" podUID="0ee11042-8133-4e65-950e-224a5b4ca910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 01 22:36:45.501105 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:45.501065 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:45.501986 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:45.501965 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:45.502891 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:36:45.502865 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 22:36:46.976860 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:36:46.975743 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:37:10.430743 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:10.430705 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:37:16.977624 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:37:16.977587 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:37:23.475846 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.475796 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz"] Nov 01 22:37:23.479977 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.479958 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:23.491096 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.491068 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz"] Nov 01 22:37:23.570124 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.570091 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-workload-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:23.570288 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.570225 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db213002-5fde-4c7a-ba7e-6762c011b4a9-istiod-ca-cert\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:23.570288 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.570259 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db213002-5fde-4c7a-ba7e-6762c011b4a9-istio-token\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:23.570402 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.570288 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpsqg\" (UniqueName: \"kubernetes.io/projected/db213002-5fde-4c7a-ba7e-6762c011b4a9-kube-api-access-zpsqg\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:23.570402 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.570311 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-credential-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:23.570484 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.570400 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-workload-certs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:23.570537 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.570516 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db213002-5fde-4c7a-ba7e-6762c011b4a9-istio-podinfo\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:23.570574 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.570555 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-istio-data\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:23.570630 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.570590 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-istio-envoy\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:23.671369 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.671328 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db213002-5fde-4c7a-ba7e-6762c011b4a9-istio-podinfo\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:23.671568 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.671395 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-istio-data\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:23.671568 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.671450 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-istio-envoy\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:23.671568 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.671520 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-workload-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:23.671763 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.671631 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db213002-5fde-4c7a-ba7e-6762c011b4a9-istiod-ca-cert\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:23.671763 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.671669 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db213002-5fde-4c7a-ba7e-6762c011b4a9-istio-token\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:23.671763 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.671710 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpsqg\" (UniqueName: \"kubernetes.io/projected/db213002-5fde-4c7a-ba7e-6762c011b4a9-kube-api-access-zpsqg\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:23.671763 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.671748 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-credential-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:23.672002 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.671771 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-workload-certs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:23.672719 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.672166 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-workload-certs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:23.672719 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.672394 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-workload-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:23.672719 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.672672 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-credential-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:23.672983 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.672811 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db213002-5fde-4c7a-ba7e-6762c011b4a9-istiod-ca-cert\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:23.673116 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.673091 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-istio-data\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:23.674309 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.674280 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-istio-envoy\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:23.675198 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.675175 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db213002-5fde-4c7a-ba7e-6762c011b4a9-istio-podinfo\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:23.675387 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.675363 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w"] Nov 01 22:37:23.679903 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.679880 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:23.682145 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.682115 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db213002-5fde-4c7a-ba7e-6762c011b4a9-istio-token\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:23.682534 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.682513 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpsqg\" (UniqueName: \"kubernetes.io/projected/db213002-5fde-4c7a-ba7e-6762c011b4a9-kube-api-access-zpsqg\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:23.698906 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.698820 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w"] Nov 01 22:37:23.773333 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.773247 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8318b18-986e-4435-8518-42d01c66b6b4-istiod-ca-cert\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:23.773333 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.773294 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-workload-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:23.773333 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.773312 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-istio-data\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:23.773558 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.773413 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-workload-certs\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:23.773558 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.773462 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-credential-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:23.773626 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.773580 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8n9c\" (UniqueName: \"kubernetes.io/projected/b8318b18-986e-4435-8518-42d01c66b6b4-kube-api-access-v8n9c\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:23.773626 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.773615 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-istio-envoy\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:23.773719 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.773644 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8318b18-986e-4435-8518-42d01c66b6b4-istio-podinfo\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:23.773791 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.773771 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8318b18-986e-4435-8518-42d01c66b6b4-istio-token\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:23.801842 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.801807 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:23.815367 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.815334 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz"] Nov 01 22:37:23.821486 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.821464 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:37:23.829378 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.829347 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz"] Nov 01 22:37:23.831464 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.831437 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-927b68\"/\"distinct-w6-927b68-kserve-self-signed-certs\"" Nov 01 22:37:23.875418 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.875389 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-istio-envoy\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:23.875534 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.875442 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8318b18-986e-4435-8518-42d01c66b6b4-istio-podinfo\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:23.875601 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.875586 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8318b18-986e-4435-8518-42d01c66b6b4-istio-token\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:23.875655 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.875623 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg9vk\" (UniqueName: \"kubernetes.io/projected/51df36c5-b9ce-4b2f-8525-5aa2d8508801-kube-api-access-kg9vk\") pod \"distinct-w6-927b68-kserve-85cd564dbc-w4dsz\" (UID: \"51df36c5-b9ce-4b2f-8525-5aa2d8508801\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:37:23.875727 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.875711 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51df36c5-b9ce-4b2f-8525-5aa2d8508801-dshm\") pod \"distinct-w6-927b68-kserve-85cd564dbc-w4dsz\" (UID: \"51df36c5-b9ce-4b2f-8525-5aa2d8508801\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:37:23.875789 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.875742 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8318b18-986e-4435-8518-42d01c66b6b4-istiod-ca-cert\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:23.875789 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.875780 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51df36c5-b9ce-4b2f-8525-5aa2d8508801-model-cache\") pod \"distinct-w6-927b68-kserve-85cd564dbc-w4dsz\" (UID: \"51df36c5-b9ce-4b2f-8525-5aa2d8508801\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:37:23.876304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.876151 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51df36c5-b9ce-4b2f-8525-5aa2d8508801-home\") pod \"distinct-w6-927b68-kserve-85cd564dbc-w4dsz\" (UID: \"51df36c5-b9ce-4b2f-8525-5aa2d8508801\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:37:23.876304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.876205 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-workload-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:23.876304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.876233 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-istio-data\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:23.876304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.876269 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-workload-certs\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:23.876304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.876329 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51df36c5-b9ce-4b2f-8525-5aa2d8508801-tls-certs\") pod \"distinct-w6-927b68-kserve-85cd564dbc-w4dsz\" (UID: \"51df36c5-b9ce-4b2f-8525-5aa2d8508801\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:37:23.876809 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.876367 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-credential-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:23.876809 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.876459 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51df36c5-b9ce-4b2f-8525-5aa2d8508801-tmp-dir\") pod \"distinct-w6-927b68-kserve-85cd564dbc-w4dsz\" (UID: \"51df36c5-b9ce-4b2f-8525-5aa2d8508801\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:37:23.876809 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.876533 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8n9c\" (UniqueName: \"kubernetes.io/projected/b8318b18-986e-4435-8518-42d01c66b6b4-kube-api-access-v8n9c\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:23.877744 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.877714 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-istio-data\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:23.877869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.877753 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8318b18-986e-4435-8518-42d01c66b6b4-istiod-ca-cert\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:23.878232 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.878166 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-credential-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:23.878389 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.878207 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-workload-certs\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:23.878502 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.878325 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-workload-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:23.878502 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.878479 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-istio-envoy\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:23.879323 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.879303 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8318b18-986e-4435-8518-42d01c66b6b4-istio-podinfo\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:23.889177 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.889149 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8n9c\" (UniqueName: \"kubernetes.io/projected/b8318b18-986e-4435-8518-42d01c66b6b4-kube-api-access-v8n9c\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:23.907923 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.907861 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8318b18-986e-4435-8518-42d01c66b6b4-istio-token\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:23.999900 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.981501 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51df36c5-b9ce-4b2f-8525-5aa2d8508801-tmp-dir\") pod \"distinct-w6-927b68-kserve-85cd564dbc-w4dsz\" (UID: \"51df36c5-b9ce-4b2f-8525-5aa2d8508801\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:37:23.999900 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.981635 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg9vk\" (UniqueName: \"kubernetes.io/projected/51df36c5-b9ce-4b2f-8525-5aa2d8508801-kube-api-access-kg9vk\") pod \"distinct-w6-927b68-kserve-85cd564dbc-w4dsz\" (UID: \"51df36c5-b9ce-4b2f-8525-5aa2d8508801\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:37:23.999900 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.981725 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51df36c5-b9ce-4b2f-8525-5aa2d8508801-dshm\") pod \"distinct-w6-927b68-kserve-85cd564dbc-w4dsz\" (UID: \"51df36c5-b9ce-4b2f-8525-5aa2d8508801\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:37:23.999900 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.981763 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51df36c5-b9ce-4b2f-8525-5aa2d8508801-model-cache\") pod \"distinct-w6-927b68-kserve-85cd564dbc-w4dsz\" (UID: \"51df36c5-b9ce-4b2f-8525-5aa2d8508801\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:37:23.999900 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.981794 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51df36c5-b9ce-4b2f-8525-5aa2d8508801-home\") pod \"distinct-w6-927b68-kserve-85cd564dbc-w4dsz\" (UID: \"51df36c5-b9ce-4b2f-8525-5aa2d8508801\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:37:23.999900 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:23.981877 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51df36c5-b9ce-4b2f-8525-5aa2d8508801-tls-certs\") pod \"distinct-w6-927b68-kserve-85cd564dbc-w4dsz\" (UID: \"51df36c5-b9ce-4b2f-8525-5aa2d8508801\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:37:24.014487 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:24.014250 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51df36c5-b9ce-4b2f-8525-5aa2d8508801-model-cache\") pod \"distinct-w6-927b68-kserve-85cd564dbc-w4dsz\" (UID: \"51df36c5-b9ce-4b2f-8525-5aa2d8508801\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:37:24.014487 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:24.003661 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51df36c5-b9ce-4b2f-8525-5aa2d8508801-tmp-dir\") pod \"distinct-w6-927b68-kserve-85cd564dbc-w4dsz\" (UID: \"51df36c5-b9ce-4b2f-8525-5aa2d8508801\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:37:24.014487 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:24.014444 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51df36c5-b9ce-4b2f-8525-5aa2d8508801-home\") pod \"distinct-w6-927b68-kserve-85cd564dbc-w4dsz\" (UID: \"51df36c5-b9ce-4b2f-8525-5aa2d8508801\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:37:24.033962 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:24.028263 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51df36c5-b9ce-4b2f-8525-5aa2d8508801-dshm\") pod \"distinct-w6-927b68-kserve-85cd564dbc-w4dsz\" (UID: \"51df36c5-b9ce-4b2f-8525-5aa2d8508801\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:37:24.033962 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:24.028285 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:24.045300 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:24.045043 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg9vk\" (UniqueName: \"kubernetes.io/projected/51df36c5-b9ce-4b2f-8525-5aa2d8508801-kube-api-access-kg9vk\") pod \"distinct-w6-927b68-kserve-85cd564dbc-w4dsz\" (UID: \"51df36c5-b9ce-4b2f-8525-5aa2d8508801\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:37:24.045497 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:24.045399 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51df36c5-b9ce-4b2f-8525-5aa2d8508801-tls-certs\") pod \"distinct-w6-927b68-kserve-85cd564dbc-w4dsz\" (UID: \"51df36c5-b9ce-4b2f-8525-5aa2d8508801\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:37:24.106140 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:24.106066 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz"] Nov 01 22:37:24.113190 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:37:24.113165 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb213002_5fde_4c7a_ba7e_6762c011b4a9.slice/crio-06251651c92740284eb773426239cad97af4168319d051cf5d1cdbc00769cfd9 WatchSource:0}: Error finding container 06251651c92740284eb773426239cad97af4168319d051cf5d1cdbc00769cfd9: Status 404 returned error can't find the container with id 06251651c92740284eb773426239cad97af4168319d051cf5d1cdbc00769cfd9 Nov 01 22:37:24.123731 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:24.123589 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:37:24.123731 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:24.123683 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:37:24.123881 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:24.123727 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:37:24.162584 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:24.162545 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:37:24.336316 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:24.336271 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w"] Nov 01 22:37:24.336527 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:37:24.336378 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8318b18_986e_4435_8518_42d01c66b6b4.slice/crio-2569002280c0338d85a4b1c96793b56365af27ba103db428645607de439771dd WatchSource:0}: Error finding container 2569002280c0338d85a4b1c96793b56365af27ba103db428645607de439771dd: Status 404 returned error can't find the container with id 2569002280c0338d85a4b1c96793b56365af27ba103db428645607de439771dd Nov 01 22:37:24.339306 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:24.339264 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:37:24.339388 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:24.339352 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:37:24.339426 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:24.339396 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:37:24.486137 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:24.486112 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz"] Nov 01 22:37:24.488379 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:37:24.488350 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51df36c5_b9ce_4b2f_8525_5aa2d8508801.slice/crio-b949a62095ead76c4fde339dd92ddcfd587976fb83f232368e60c44202e6122d WatchSource:0}: Error finding container b949a62095ead76c4fde339dd92ddcfd587976fb83f232368e60c44202e6122d: Status 404 returned error can't find the container with id b949a62095ead76c4fde339dd92ddcfd587976fb83f232368e60c44202e6122d Nov 01 22:37:25.106946 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:25.106909 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" event={"ID":"51df36c5-b9ce-4b2f-8525-5aa2d8508801","Type":"ContainerStarted","Data":"4788ab542bd84628d8b3a1c6cbbfd4af93208e2341585140995d0fd03c14c5b5"} Nov 01 22:37:25.107512 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:25.107485 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" event={"ID":"51df36c5-b9ce-4b2f-8525-5aa2d8508801","Type":"ContainerStarted","Data":"b949a62095ead76c4fde339dd92ddcfd587976fb83f232368e60c44202e6122d"} Nov 01 22:37:25.109907 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:25.109579 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" event={"ID":"b8318b18-986e-4435-8518-42d01c66b6b4","Type":"ContainerStarted","Data":"e25506f889102a6e7a09d747d53bf50317ddb28962d1b084f9b1b1103c73a583"} Nov 01 22:37:25.109907 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:25.109639 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" event={"ID":"b8318b18-986e-4435-8518-42d01c66b6b4","Type":"ContainerStarted","Data":"2569002280c0338d85a4b1c96793b56365af27ba103db428645607de439771dd"} Nov 01 22:37:25.111549 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:25.111526 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" event={"ID":"db213002-5fde-4c7a-ba7e-6762c011b4a9","Type":"ContainerStarted","Data":"3f59012b57498e6be5618e057870b6a1110f62f48568e383f9a69ab8aecb38f1"} Nov 01 22:37:25.111674 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:25.111589 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" event={"ID":"db213002-5fde-4c7a-ba7e-6762c011b4a9","Type":"ContainerStarted","Data":"06251651c92740284eb773426239cad97af4168319d051cf5d1cdbc00769cfd9"} Nov 01 22:37:25.127847 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:25.127785 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podStartSLOduration=2.127773121 podStartE2EDuration="2.127773121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:25.126952739 +0000 UTC m=+813.531543363" watchObservedRunningTime="2026-11-01 22:37:25.127773121 +0000 UTC m=+813.532363753" Nov 01 22:37:25.174460 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:25.174420 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:37:25.201086 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:25.201025 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" podStartSLOduration=2.201008698 podStartE2EDuration="2.201008698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:25.200415559 +0000 UTC m=+813.605006192" watchObservedRunningTime="2026-11-01 22:37:25.201008698 +0000 UTC m=+813.605599331" Nov 01 22:37:25.204824 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:25.204752 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" podStartSLOduration=2.204734272 podStartE2EDuration="2.204734272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:25.174247567 +0000 UTC m=+813.578838200" watchObservedRunningTime="2026-11-01 22:37:25.204734272 +0000 UTC m=+813.609324905" Nov 01 22:37:25.809350 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:25.809070 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 01 22:37:25.809350 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:25.809153 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" podUID="db213002-5fde-4c7a-ba7e-6762c011b4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 01 22:37:25.810415 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:25.810349 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:26.030856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:26.030251 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:26.049858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:26.039969 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:37:26.049858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:26.040029 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" podUID="b8318b18-986e-4435-8518-42d01c66b6b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 22:37:26.805414 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:26.805377 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 01 22:37:26.805678 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:26.805650 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" podUID="db213002-5fde-4c7a-ba7e-6762c011b4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 01 22:37:27.043308 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:27.043268 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:37:27.043931 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:27.043886 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" podUID="b8318b18-986e-4435-8518-42d01c66b6b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 22:37:27.807749 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:27.807716 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 01 22:37:27.808071 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:27.808043 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" podUID="db213002-5fde-4c7a-ba7e-6762c011b4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 01 22:37:27.861858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:27.848940 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:28.034857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:28.029290 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:37:28.034857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:28.029358 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" podUID="b8318b18-986e-4435-8518-42d01c66b6b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 22:37:28.345014 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:28.343942 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:28.475023 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:28.474964 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:28.804858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:28.804169 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 01 22:37:28.804858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:28.804230 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" podUID="db213002-5fde-4c7a-ba7e-6762c011b4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 01 22:37:29.030856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:29.030619 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:37:29.030856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:29.030681 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" podUID="b8318b18-986e-4435-8518-42d01c66b6b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 22:37:29.301343 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:29.280354 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:29.357587 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:29.357537 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:37:29.371710 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:29.371683 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:37:29.423905 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:29.414746 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:29.681691 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:29.678915 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:37:29.681691 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:29.681461 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:37:29.720011 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:29.719956 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:30.309292 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:30.309250 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:37:30.330218 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:30.330171 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" podUID="b8318b18-986e-4435-8518-42d01c66b6b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 22:37:30.358404 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:30.358369 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 01 22:37:30.375681 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:30.375623 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" podUID="db213002-5fde-4c7a-ba7e-6762c011b4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 01 22:37:30.417662 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:30.417617 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:37:30.506851 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:30.501923 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:37:30.548085 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:30.548051 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:37:30.687887 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:30.677020 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:30.687887 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:30.678231 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:37:30.687887 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:30.680640 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:30.834487 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:30.831249 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 01 22:37:30.834487 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:30.831328 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" podUID="db213002-5fde-4c7a-ba7e-6762c011b4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 01 22:37:31.058594 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:31.057180 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:37:31.058594 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:31.057247 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" podUID="b8318b18-986e-4435-8518-42d01c66b6b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 22:37:31.464934 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:31.438336 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:31.493856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:31.457541 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:37:31.523020 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:31.522957 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:31.681981 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:31.681934 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:37:31.823494 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:31.823378 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 01 22:37:31.823777 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:31.823724 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" podUID="db213002-5fde-4c7a-ba7e-6762c011b4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 01 22:37:32.038569 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:32.038535 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:37:32.038871 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:32.038812 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" podUID="b8318b18-986e-4435-8518-42d01c66b6b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 22:37:32.342375 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:32.342346 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:37:32.418761 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:32.417454 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:37:32.418761 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:32.418153 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:37:32.809855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:32.809268 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 01 22:37:32.809855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:32.809332 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" podUID="db213002-5fde-4c7a-ba7e-6762c011b4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 01 22:37:33.029266 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:33.029229 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 22:37:33.029553 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:33.029524 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" podUID="b8318b18-986e-4435-8518-42d01c66b6b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 22:37:34.163824 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:34.163784 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:37:34.178310 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:34.178278 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:37:34.773653 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:34.773614 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:34.774092 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:34.774067 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:34.794108 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:34.794076 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 22:37:35.018257 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:35.018224 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:35.019540 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:35.019513 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:35.021778 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:35.021753 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:37:35.036322 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:35.036246 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:37:37.919252 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:37.919213 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc"] Nov 01 22:37:37.923476 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:37.923453 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:37.931714 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:37.931697 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-2f3705-openshift-default-dockercfg-29c4c\"" Nov 01 22:37:37.942422 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:37.942396 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc"] Nov 01 22:37:37.989561 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:37.989521 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-credential-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:37.989561 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:37.989564 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-workload-certs\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:37.989811 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:37.989582 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istio-data\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:37.989811 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:37.989736 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istiod-ca-cert\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:37.989950 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:37.989884 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-workload-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:37.989950 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:37.989910 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istio-token\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:37.990039 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:37.990015 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8tnl\" (UniqueName: \"kubernetes.io/projected/7e14ab9c-363b-4f66-a585-3875dfaa45b6-kube-api-access-z8tnl\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:37.990080 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:37.990050 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istio-podinfo\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:37.990080 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:37.990072 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istio-envoy\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:38.090746 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:38.090709 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8tnl\" (UniqueName: \"kubernetes.io/projected/7e14ab9c-363b-4f66-a585-3875dfaa45b6-kube-api-access-z8tnl\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:38.090959 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:38.090760 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istio-podinfo\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:38.090959 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:38.090808 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istio-envoy\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:38.090959 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:38.090854 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-credential-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:38.090959 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:38.090883 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-workload-certs\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:38.090959 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:38.090905 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istio-data\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:38.091199 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:38.090968 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istiod-ca-cert\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:38.091255 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:38.091189 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-workload-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:38.091255 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:38.091237 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istio-token\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:38.091391 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:38.091356 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-workload-certs\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:38.091452 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:38.091410 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istio-data\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:38.091896 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:38.091868 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-workload-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:38.092164 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:38.092142 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-credential-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:38.092268 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:38.092246 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istiod-ca-cert\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:38.093606 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:38.093580 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istio-envoy\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:38.093706 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:38.093651 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istio-podinfo\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:38.102415 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:38.102392 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istio-token\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:38.102552 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:38.102534 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8tnl\" (UniqueName: \"kubernetes.io/projected/7e14ab9c-363b-4f66-a585-3875dfaa45b6-kube-api-access-z8tnl\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:38.237380 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:38.237349 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:38.525064 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:38.525021 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc"] Nov 01 22:37:38.526978 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:37:38.526946 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e14ab9c_363b_4f66_a585_3875dfaa45b6.slice/crio-efed7206ef7242fc5fce1be79e73ab1b59a8c4642b31d59ba69eceadd8ddd7ee WatchSource:0}: Error finding container efed7206ef7242fc5fce1be79e73ab1b59a8c4642b31d59ba69eceadd8ddd7ee: Status 404 returned error can't find the container with id efed7206ef7242fc5fce1be79e73ab1b59a8c4642b31d59ba69eceadd8ddd7ee Nov 01 22:37:38.529626 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:38.529588 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:37:38.529738 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:38.529679 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:37:38.529738 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:38.529715 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:37:39.053620 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:39.053578 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" event={"ID":"7e14ab9c-363b-4f66-a585-3875dfaa45b6","Type":"ContainerStarted","Data":"20cef8e5a3ffb47b4b73b9dfa5f58408b1d8d7588eac30b4d72e67b0fb65de28"} Nov 01 22:37:39.054312 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:39.054284 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" event={"ID":"7e14ab9c-363b-4f66-a585-3875dfaa45b6","Type":"ContainerStarted","Data":"efed7206ef7242fc5fce1be79e73ab1b59a8c4642b31d59ba69eceadd8ddd7ee"} Nov 01 22:37:39.086305 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:39.086248 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" podStartSLOduration=2.086230159 podStartE2EDuration="2.086230159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:39.085352229 +0000 UTC m=+827.489942886" watchObservedRunningTime="2026-11-01 22:37:39.086230159 +0000 UTC m=+827.490820791" Nov 01 22:37:39.238466 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:39.238434 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:39.239952 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:39.239926 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 01 22:37:39.240094 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:39.239980 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" podUID="7e14ab9c-363b-4f66-a585-3875dfaa45b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 01 22:37:40.237958 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:40.237924 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 01 22:37:40.238371 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:40.237991 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" podUID="7e14ab9c-363b-4f66-a585-3875dfaa45b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 01 22:37:41.238131 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:41.238092 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 01 22:37:41.238549 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:41.238157 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" podUID="7e14ab9c-363b-4f66-a585-3875dfaa45b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 01 22:37:42.237670 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:42.237636 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 01 22:37:42.237877 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:42.237696 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" podUID="7e14ab9c-363b-4f66-a585-3875dfaa45b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 01 22:37:43.238094 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:43.238057 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 01 22:37:43.238470 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:43.238130 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" podUID="7e14ab9c-363b-4f66-a585-3875dfaa45b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 01 22:37:44.237669 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:44.237630 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 01 22:37:44.237890 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:44.237690 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" podUID="7e14ab9c-363b-4f66-a585-3875dfaa45b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 01 22:37:44.807926 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:44.807887 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:37:45.732857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:45.732402 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:45.737978 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:45.734212 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:45.737978 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:45.736013 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 22:37:46.628857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:46.627896 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n"] Nov 01 22:37:46.720868 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:46.709274 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n"] Nov 01 22:37:46.743891 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:46.713336 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:37:46.830863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:46.830297 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99d6cd19-de6a-43ff-8a40-5df59db6818c-tmp-dir\") pod \"distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n\" (UID: \"99d6cd19-de6a-43ff-8a40-5df59db6818c\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:37:46.830863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:46.830398 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99d6cd19-de6a-43ff-8a40-5df59db6818c-dshm\") pod \"distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n\" (UID: \"99d6cd19-de6a-43ff-8a40-5df59db6818c\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:37:46.830863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:46.830427 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx6j7\" (UniqueName: \"kubernetes.io/projected/99d6cd19-de6a-43ff-8a40-5df59db6818c-kube-api-access-xx6j7\") pod \"distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n\" (UID: \"99d6cd19-de6a-43ff-8a40-5df59db6818c\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:37:46.830863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:46.830542 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99d6cd19-de6a-43ff-8a40-5df59db6818c-model-cache\") pod \"distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n\" (UID: \"99d6cd19-de6a-43ff-8a40-5df59db6818c\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:37:46.830863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:46.830578 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99d6cd19-de6a-43ff-8a40-5df59db6818c-tls-certs\") pod \"distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n\" (UID: \"99d6cd19-de6a-43ff-8a40-5df59db6818c\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:37:46.830863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:46.830626 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99d6cd19-de6a-43ff-8a40-5df59db6818c-home\") pod \"distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n\" (UID: \"99d6cd19-de6a-43ff-8a40-5df59db6818c\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:37:46.835197 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:46.834740 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f338e4\"/\"distinct-2-w1-f338e4-kserve-self-signed-certs\"" Nov 01 22:37:46.938911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:46.934433 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99d6cd19-de6a-43ff-8a40-5df59db6818c-tmp-dir\") pod \"distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n\" (UID: \"99d6cd19-de6a-43ff-8a40-5df59db6818c\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:37:46.938911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:46.934532 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99d6cd19-de6a-43ff-8a40-5df59db6818c-dshm\") pod \"distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n\" (UID: \"99d6cd19-de6a-43ff-8a40-5df59db6818c\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:37:46.938911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:46.934592 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx6j7\" (UniqueName: \"kubernetes.io/projected/99d6cd19-de6a-43ff-8a40-5df59db6818c-kube-api-access-xx6j7\") pod \"distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n\" (UID: \"99d6cd19-de6a-43ff-8a40-5df59db6818c\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:37:46.938911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:46.934722 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99d6cd19-de6a-43ff-8a40-5df59db6818c-model-cache\") pod \"distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n\" (UID: \"99d6cd19-de6a-43ff-8a40-5df59db6818c\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:37:46.938911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:46.934761 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99d6cd19-de6a-43ff-8a40-5df59db6818c-tls-certs\") pod \"distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n\" (UID: \"99d6cd19-de6a-43ff-8a40-5df59db6818c\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:37:46.938911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:46.934817 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99d6cd19-de6a-43ff-8a40-5df59db6818c-home\") pod \"distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n\" (UID: \"99d6cd19-de6a-43ff-8a40-5df59db6818c\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:37:46.938911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:46.936505 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99d6cd19-de6a-43ff-8a40-5df59db6818c-model-cache\") pod \"distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n\" (UID: \"99d6cd19-de6a-43ff-8a40-5df59db6818c\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:37:46.938911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:46.936754 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99d6cd19-de6a-43ff-8a40-5df59db6818c-home\") pod \"distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n\" (UID: \"99d6cd19-de6a-43ff-8a40-5df59db6818c\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:37:46.944947 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:46.944903 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99d6cd19-de6a-43ff-8a40-5df59db6818c-tmp-dir\") pod \"distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n\" (UID: \"99d6cd19-de6a-43ff-8a40-5df59db6818c\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:37:46.965567 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:46.965536 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99d6cd19-de6a-43ff-8a40-5df59db6818c-dshm\") pod \"distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n\" (UID: \"99d6cd19-de6a-43ff-8a40-5df59db6818c\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:37:46.969952 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:46.969896 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx6j7\" (UniqueName: \"kubernetes.io/projected/99d6cd19-de6a-43ff-8a40-5df59db6818c-kube-api-access-xx6j7\") pod \"distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n\" (UID: \"99d6cd19-de6a-43ff-8a40-5df59db6818c\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:37:46.989316 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:46.987590 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99d6cd19-de6a-43ff-8a40-5df59db6818c-tls-certs\") pod \"distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n\" (UID: \"99d6cd19-de6a-43ff-8a40-5df59db6818c\") " pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:37:47.145856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:47.143259 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:37:47.266653 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:37:47.266611 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:37:47.727866 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:47.727805 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n"] Nov 01 22:37:48.227055 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:48.227020 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" event={"ID":"99d6cd19-de6a-43ff-8a40-5df59db6818c","Type":"ContainerStarted","Data":"8393223ff6c758f1b996c2ab2d2277dab50ec0fee24b9ea57e54df34b7f1c8d2"} Nov 01 22:37:48.227241 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:48.227062 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" event={"ID":"99d6cd19-de6a-43ff-8a40-5df59db6818c","Type":"ContainerStarted","Data":"c65b0d33ffbe5063c74ae890faddcb4ced268bbcbc97dcef18b9aaa874c102a2"} Nov 01 22:37:48.248752 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:48.248699 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podStartSLOduration=2.248684009 podStartE2EDuration="2.248684009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:48.245312984 +0000 UTC m=+836.649903626" watchObservedRunningTime="2026-11-01 22:37:48.248684009 +0000 UTC m=+836.653274640" Nov 01 22:37:49.144357 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:49.144321 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:37:54.517588 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:54.517552 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4"] Nov 01 22:37:54.522092 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:54.522070 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:37:54.530307 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:54.530282 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-e26733\"/\"openshift-service-ca.crt\"" Nov 01 22:37:54.530943 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:54.530330 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e26733\"/\"distinct-w5-e26733-kserve-self-signed-certs\"" Nov 01 22:37:54.530943 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:54.530372 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-e26733\"/\"kube-root-ca.crt\"" Nov 01 22:37:54.530943 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:54.530723 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e26733\"/\"default-dockercfg-k6rcx\"" Nov 01 22:37:54.531624 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:54.531607 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4"] Nov 01 22:37:54.615941 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:54.615904 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-tls-certs\") pod \"distinct-w5-e26733-kserve-6c8548bbdb-lddj4\" (UID: \"0a41f431-3a44-4fff-9b38-ad4e52c4fe33\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:37:54.616152 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:54.616067 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-model-cache\") pod \"distinct-w5-e26733-kserve-6c8548bbdb-lddj4\" (UID: \"0a41f431-3a44-4fff-9b38-ad4e52c4fe33\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:37:54.616152 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:54.616095 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztvkx\" (UniqueName: \"kubernetes.io/projected/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-kube-api-access-ztvkx\") pod \"distinct-w5-e26733-kserve-6c8548bbdb-lddj4\" (UID: \"0a41f431-3a44-4fff-9b38-ad4e52c4fe33\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:37:54.616288 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:54.616262 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-home\") pod \"distinct-w5-e26733-kserve-6c8548bbdb-lddj4\" (UID: \"0a41f431-3a44-4fff-9b38-ad4e52c4fe33\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:37:54.616330 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:54.616295 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-dshm\") pod \"distinct-w5-e26733-kserve-6c8548bbdb-lddj4\" (UID: \"0a41f431-3a44-4fff-9b38-ad4e52c4fe33\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:37:54.616330 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:54.616325 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-tmp-dir\") pod \"distinct-w5-e26733-kserve-6c8548bbdb-lddj4\" (UID: \"0a41f431-3a44-4fff-9b38-ad4e52c4fe33\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:37:54.725548 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:54.722634 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-home\") pod \"distinct-w5-e26733-kserve-6c8548bbdb-lddj4\" (UID: \"0a41f431-3a44-4fff-9b38-ad4e52c4fe33\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:37:54.725548 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:54.722688 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-dshm\") pod \"distinct-w5-e26733-kserve-6c8548bbdb-lddj4\" (UID: \"0a41f431-3a44-4fff-9b38-ad4e52c4fe33\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:37:54.725548 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:54.722737 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-tmp-dir\") pod \"distinct-w5-e26733-kserve-6c8548bbdb-lddj4\" (UID: \"0a41f431-3a44-4fff-9b38-ad4e52c4fe33\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:37:54.725548 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:54.722798 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-tls-certs\") pod \"distinct-w5-e26733-kserve-6c8548bbdb-lddj4\" (UID: \"0a41f431-3a44-4fff-9b38-ad4e52c4fe33\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:37:54.725548 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:54.722982 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-model-cache\") pod \"distinct-w5-e26733-kserve-6c8548bbdb-lddj4\" (UID: \"0a41f431-3a44-4fff-9b38-ad4e52c4fe33\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:37:54.725548 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:54.723010 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztvkx\" (UniqueName: \"kubernetes.io/projected/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-kube-api-access-ztvkx\") pod \"distinct-w5-e26733-kserve-6c8548bbdb-lddj4\" (UID: \"0a41f431-3a44-4fff-9b38-ad4e52c4fe33\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:37:54.725548 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:54.723873 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-home\") pod \"distinct-w5-e26733-kserve-6c8548bbdb-lddj4\" (UID: \"0a41f431-3a44-4fff-9b38-ad4e52c4fe33\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:37:54.725548 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:54.725140 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-tmp-dir\") pod \"distinct-w5-e26733-kserve-6c8548bbdb-lddj4\" (UID: \"0a41f431-3a44-4fff-9b38-ad4e52c4fe33\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:37:54.725548 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:54.725355 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-model-cache\") pod \"distinct-w5-e26733-kserve-6c8548bbdb-lddj4\" (UID: \"0a41f431-3a44-4fff-9b38-ad4e52c4fe33\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:37:54.748511 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:54.748475 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-tls-certs\") pod \"distinct-w5-e26733-kserve-6c8548bbdb-lddj4\" (UID: \"0a41f431-3a44-4fff-9b38-ad4e52c4fe33\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:37:54.762514 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:54.762482 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztvkx\" (UniqueName: \"kubernetes.io/projected/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-kube-api-access-ztvkx\") pod \"distinct-w5-e26733-kserve-6c8548bbdb-lddj4\" (UID: \"0a41f431-3a44-4fff-9b38-ad4e52c4fe33\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:37:54.763037 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:54.762953 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-dshm\") pod \"distinct-w5-e26733-kserve-6c8548bbdb-lddj4\" (UID: \"0a41f431-3a44-4fff-9b38-ad4e52c4fe33\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:37:54.837462 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:54.837380 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:37:55.120451 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:55.120416 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4"] Nov 01 22:37:55.129154 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:37:55.124911 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a41f431_3a44_4fff_9b38_ad4e52c4fe33.slice/crio-462e24422b9a5fe7181f7619fddcbdf0532ab93eeceffea2fd75d15243c723d5 WatchSource:0}: Error finding container 462e24422b9a5fe7181f7619fddcbdf0532ab93eeceffea2fd75d15243c723d5: Status 404 returned error can't find the container with id 462e24422b9a5fe7181f7619fddcbdf0532ab93eeceffea2fd75d15243c723d5 Nov 01 22:37:55.285619 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:55.285580 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" event={"ID":"0a41f431-3a44-4fff-9b38-ad4e52c4fe33","Type":"ContainerStarted","Data":"83bbb81a37f8939a60fe1bd91c6130afd7bf90a8645009482ce904108b9f8f38"} Nov 01 22:37:55.285806 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:55.285627 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" event={"ID":"0a41f431-3a44-4fff-9b38-ad4e52c4fe33","Type":"ContainerStarted","Data":"462e24422b9a5fe7181f7619fddcbdf0532ab93eeceffea2fd75d15243c723d5"} Nov 01 22:37:55.306868 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:55.306800 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podStartSLOduration=1.306785648 podStartE2EDuration="1.306785648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:55.30476448 +0000 UTC m=+843.709355110" watchObservedRunningTime="2026-11-01 22:37:55.306785648 +0000 UTC m=+843.711376257" Nov 01 22:37:55.838296 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:55.838262 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:37:57.144281 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:57.144243 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:37:57.161454 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:57.161419 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:37:57.309672 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:37:57.309638 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:38:02.032860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:02.032673 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t"] Nov 01 22:38:02.051141 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:02.051110 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:38:02.061455 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:02.061420 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e5f47e\"/\"distinct-2-w0-e5f47e-kserve-self-signed-certs\"" Nov 01 22:38:02.067864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:02.067108 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t"] Nov 01 22:38:02.120858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:02.104266 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56eccabb-80c5-4796-bc93-f07a18bb160a-tmp-dir\") pod \"distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t\" (UID: \"56eccabb-80c5-4796-bc93-f07a18bb160a\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:38:02.121181 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:02.121151 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56eccabb-80c5-4796-bc93-f07a18bb160a-tls-certs\") pod \"distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t\" (UID: \"56eccabb-80c5-4796-bc93-f07a18bb160a\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:38:02.169923 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:02.124940 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56eccabb-80c5-4796-bc93-f07a18bb160a-home\") pod \"distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t\" (UID: \"56eccabb-80c5-4796-bc93-f07a18bb160a\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:38:02.170266 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:02.170229 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg2v5\" (UniqueName: \"kubernetes.io/projected/56eccabb-80c5-4796-bc93-f07a18bb160a-kube-api-access-sg2v5\") pod \"distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t\" (UID: \"56eccabb-80c5-4796-bc93-f07a18bb160a\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:38:02.170459 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:02.170441 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56eccabb-80c5-4796-bc93-f07a18bb160a-dshm\") pod \"distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t\" (UID: \"56eccabb-80c5-4796-bc93-f07a18bb160a\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:38:02.170677 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:02.170659 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56eccabb-80c5-4796-bc93-f07a18bb160a-model-cache\") pod \"distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t\" (UID: \"56eccabb-80c5-4796-bc93-f07a18bb160a\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:38:02.307065 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:02.276113 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56eccabb-80c5-4796-bc93-f07a18bb160a-tmp-dir\") pod \"distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t\" (UID: \"56eccabb-80c5-4796-bc93-f07a18bb160a\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:38:02.307065 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:02.276173 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56eccabb-80c5-4796-bc93-f07a18bb160a-tls-certs\") pod \"distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t\" (UID: \"56eccabb-80c5-4796-bc93-f07a18bb160a\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:38:02.307065 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:02.276238 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56eccabb-80c5-4796-bc93-f07a18bb160a-home\") pod \"distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t\" (UID: \"56eccabb-80c5-4796-bc93-f07a18bb160a\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:38:02.307065 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:02.276278 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg2v5\" (UniqueName: \"kubernetes.io/projected/56eccabb-80c5-4796-bc93-f07a18bb160a-kube-api-access-sg2v5\") pod \"distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t\" (UID: \"56eccabb-80c5-4796-bc93-f07a18bb160a\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:38:02.307065 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:02.276333 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56eccabb-80c5-4796-bc93-f07a18bb160a-dshm\") pod \"distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t\" (UID: \"56eccabb-80c5-4796-bc93-f07a18bb160a\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:38:02.307065 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:02.276441 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56eccabb-80c5-4796-bc93-f07a18bb160a-model-cache\") pod \"distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t\" (UID: \"56eccabb-80c5-4796-bc93-f07a18bb160a\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:38:02.307065 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:02.277119 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56eccabb-80c5-4796-bc93-f07a18bb160a-model-cache\") pod \"distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t\" (UID: \"56eccabb-80c5-4796-bc93-f07a18bb160a\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:38:02.307065 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:02.277606 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56eccabb-80c5-4796-bc93-f07a18bb160a-tmp-dir\") pod \"distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t\" (UID: \"56eccabb-80c5-4796-bc93-f07a18bb160a\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:38:02.307065 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:02.288858 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56eccabb-80c5-4796-bc93-f07a18bb160a-home\") pod \"distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t\" (UID: \"56eccabb-80c5-4796-bc93-f07a18bb160a\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:38:02.361846 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:02.361732 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56eccabb-80c5-4796-bc93-f07a18bb160a-tls-certs\") pod \"distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t\" (UID: \"56eccabb-80c5-4796-bc93-f07a18bb160a\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:38:02.659267 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:02.659183 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56eccabb-80c5-4796-bc93-f07a18bb160a-dshm\") pod \"distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t\" (UID: \"56eccabb-80c5-4796-bc93-f07a18bb160a\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:38:02.741858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:02.721533 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg2v5\" (UniqueName: \"kubernetes.io/projected/56eccabb-80c5-4796-bc93-f07a18bb160a-kube-api-access-sg2v5\") pod \"distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t\" (UID: \"56eccabb-80c5-4796-bc93-f07a18bb160a\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:38:02.774856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:02.769533 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:38:03.288870 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:03.286899 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t"] Nov 01 22:38:03.378916 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:03.378875 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" event={"ID":"56eccabb-80c5-4796-bc93-f07a18bb160a","Type":"ContainerStarted","Data":"ce1106520e16408ce876d1df8eba3d89f3d8f40031cb5adc12982217d1cd912c"} Nov 01 22:38:04.386694 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:04.386647 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" event={"ID":"56eccabb-80c5-4796-bc93-f07a18bb160a","Type":"ContainerStarted","Data":"4f6b26b520e35f7be79e0cf120802d6d6dffbb1a8d003e21750e2a493aa8601e"} Nov 01 22:38:04.408165 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:04.408116 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podStartSLOduration=2.408101056 podStartE2EDuration="2.408101056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:04.405215658 +0000 UTC m=+852.809806290" watchObservedRunningTime="2026-11-01 22:38:04.408101056 +0000 UTC m=+852.812691738" Nov 01 22:38:04.770415 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:04.770384 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:38:04.840941 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:04.840903 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:38:04.877202 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:04.877165 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:38:05.404014 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:05.403988 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:38:08.229742 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.229706 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb"] Nov 01 22:38:08.235194 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.235167 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:08.246051 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.245919 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths"] Nov 01 22:38:08.250519 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.250495 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:08.260486 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.260455 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb"] Nov 01 22:38:08.275256 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.275229 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths"] Nov 01 22:38:08.298427 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.298392 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istio-envoy\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:08.298641 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.298448 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs8p9\" (UniqueName: \"kubernetes.io/projected/a6d034ba-eb48-4ce8-b349-e2e816a123c1-kube-api-access-hs8p9\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:08.298641 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.298499 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istio-data\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:08.298641 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.298565 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istio-token\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:08.298641 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.298610 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-credential-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:08.298641 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.298637 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlw9t\" (UniqueName: \"kubernetes.io/projected/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-kube-api-access-nlw9t\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:08.298944 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.298691 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istio-podinfo\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:08.298944 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.298729 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-credential-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:08.298944 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.298868 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istio-podinfo\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:08.299046 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.299018 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istiod-ca-cert\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:08.299096 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.299086 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istio-envoy\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:08.299147 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.299119 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istiod-ca-cert\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:08.299208 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.299179 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-workload-certs\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:08.299265 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.299229 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istio-data\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:08.299320 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.299279 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istio-token\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:08.299433 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.299375 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-workload-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:08.299498 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.299436 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-workload-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:08.299498 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.299453 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-workload-certs\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:08.400900 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.400863 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istiod-ca-cert\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:08.401102 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.400921 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istio-envoy\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:08.401102 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.400958 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istiod-ca-cert\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:08.401102 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.400988 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-workload-certs\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:08.401102 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.401048 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istio-data\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:08.401335 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.401112 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istio-token\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:08.401335 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.401163 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-workload-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:08.401335 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.401199 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-workload-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:08.401335 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.401222 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-workload-certs\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:08.401335 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.401295 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istio-envoy\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:08.401335 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.401334 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs8p9\" (UniqueName: \"kubernetes.io/projected/a6d034ba-eb48-4ce8-b349-e2e816a123c1-kube-api-access-hs8p9\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:08.404940 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.401358 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istio-data\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:08.404940 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.401400 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istio-data\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:08.404940 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.401403 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istio-token\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:08.404940 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.401455 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-credential-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:08.404940 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.401481 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlw9t\" (UniqueName: \"kubernetes.io/projected/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-kube-api-access-nlw9t\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:08.404940 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.401537 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istio-podinfo\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:08.404940 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.401574 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-credential-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:08.404940 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.401599 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istio-podinfo\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:08.404940 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.401676 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-credential-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:08.404940 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.401475 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-workload-certs\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:08.404940 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.401737 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istiod-ca-cert\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:08.404940 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.401745 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istiod-ca-cert\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:08.404940 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.401906 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-workload-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:08.404940 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.402156 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istio-data\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:08.404940 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.402327 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-credential-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:08.404940 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.402381 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-workload-certs\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:08.404940 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.402405 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-workload-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:08.404940 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.404878 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istio-envoy\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:08.404940 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.404904 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istio-envoy\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:08.405528 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.405019 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istio-podinfo\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:08.405528 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.405327 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istio-podinfo\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:08.410824 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.410797 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istio-token\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:08.411025 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.410998 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs8p9\" (UniqueName: \"kubernetes.io/projected/a6d034ba-eb48-4ce8-b349-e2e816a123c1-kube-api-access-hs8p9\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:08.411216 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.411194 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlw9t\" (UniqueName: \"kubernetes.io/projected/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-kube-api-access-nlw9t\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:08.411348 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.411332 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istio-token\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:08.565128 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.565023 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:08.574393 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.574361 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:08.902185 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.902155 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb"] Nov 01 22:38:08.903056 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:38:08.903034 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d034ba_eb48_4ce8_b349_e2e816a123c1.slice/crio-b951ff2d2e1718deb990260b6e34bc5e0e58bb44b90583b9923f9068c255aa61 WatchSource:0}: Error finding container b951ff2d2e1718deb990260b6e34bc5e0e58bb44b90583b9923f9068c255aa61: Status 404 returned error can't find the container with id b951ff2d2e1718deb990260b6e34bc5e0e58bb44b90583b9923f9068c255aa61 Nov 01 22:38:08.905480 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.905448 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:38:08.905556 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.905514 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:38:08.905556 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.905542 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:38:08.979991 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.979963 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths"] Nov 01 22:38:08.982632 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:38:08.982592 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ec80ece_abb9_45b5_a34e_45bbb2d90eb0.slice/crio-d86d77149130a2beade216808224552d4aa0ade74eff20df26f97364b9720814 WatchSource:0}: Error finding container d86d77149130a2beade216808224552d4aa0ade74eff20df26f97364b9720814: Status 404 returned error can't find the container with id d86d77149130a2beade216808224552d4aa0ade74eff20df26f97364b9720814 Nov 01 22:38:08.985383 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.985348 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:38:08.985538 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.985456 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:38:08.985538 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:08.985529 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:38:09.424100 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:09.424067 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" event={"ID":"a6d034ba-eb48-4ce8-b349-e2e816a123c1","Type":"ContainerStarted","Data":"638eec0f18190e29198806560afe5353c4525f0bed2548d2f6c726f4bf08d87b"} Nov 01 22:38:09.424546 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:09.424126 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" event={"ID":"a6d034ba-eb48-4ce8-b349-e2e816a123c1","Type":"ContainerStarted","Data":"b951ff2d2e1718deb990260b6e34bc5e0e58bb44b90583b9923f9068c255aa61"} Nov 01 22:38:09.425929 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:09.425892 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" event={"ID":"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0","Type":"ContainerStarted","Data":"597fb365768a1a9a21f14c095594d7b749f20942526a54db7ef42a6e548f68f1"} Nov 01 22:38:09.426059 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:09.425945 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" event={"ID":"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0","Type":"ContainerStarted","Data":"d86d77149130a2beade216808224552d4aa0ade74eff20df26f97364b9720814"} Nov 01 22:38:09.446991 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:09.446937 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" podStartSLOduration=1.446922979 podStartE2EDuration="1.446922979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:09.443560176 +0000 UTC m=+857.848150810" watchObservedRunningTime="2026-11-01 22:38:09.446922979 +0000 UTC m=+857.851513611" Nov 01 22:38:09.465900 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:09.465824 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" podStartSLOduration=1.465810112 podStartE2EDuration="1.465810112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:09.46376874 +0000 UTC m=+857.868359371" watchObservedRunningTime="2026-11-01 22:38:09.465810112 +0000 UTC m=+857.870400744" Nov 01 22:38:09.571855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:09.567194 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:09.571855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:09.570966 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 22:38:09.571855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:09.571023 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" podUID="a6d034ba-eb48-4ce8-b349-e2e816a123c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 22:38:09.576864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:09.575907 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:10.567351 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:10.567309 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 22:38:10.568434 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:10.567974 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" podUID="a6d034ba-eb48-4ce8-b349-e2e816a123c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 22:38:10.593849 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:10.593778 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 01 22:38:10.594038 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:10.593888 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" podUID="5ec80ece-abb9-45b5-a34e-45bbb2d90eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 01 22:38:11.565752 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:11.565716 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 22:38:11.566118 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:11.566087 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" podUID="a6d034ba-eb48-4ce8-b349-e2e816a123c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 22:38:11.577953 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:11.577921 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 01 22:38:11.578530 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:11.578492 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" podUID="5ec80ece-abb9-45b5-a34e-45bbb2d90eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 01 22:38:12.568856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:12.568245 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 22:38:12.568856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:12.568315 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" podUID="a6d034ba-eb48-4ce8-b349-e2e816a123c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 22:38:12.585962 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:12.577253 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 01 22:38:12.585962 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:12.577310 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" podUID="5ec80ece-abb9-45b5-a34e-45bbb2d90eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 01 22:38:12.787859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:12.780259 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:38:12.803293 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:12.803266 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:38:13.509513 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:13.509484 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:38:13.569246 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:13.569211 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 22:38:13.569528 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:13.569499 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" podUID="a6d034ba-eb48-4ce8-b349-e2e816a123c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 22:38:13.596860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:13.583675 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 01 22:38:13.600481 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:13.600447 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" podUID="5ec80ece-abb9-45b5-a34e-45bbb2d90eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 01 22:38:14.567314 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:14.567272 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 22:38:14.567608 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:14.567576 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" podUID="a6d034ba-eb48-4ce8-b349-e2e816a123c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 22:38:14.594009 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:14.593966 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 01 22:38:14.594234 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:14.594042 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" podUID="5ec80ece-abb9-45b5-a34e-45bbb2d90eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 01 22:38:15.569671 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:15.568244 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 22:38:15.569671 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:15.568319 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" podUID="a6d034ba-eb48-4ce8-b349-e2e816a123c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 22:38:15.575406 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:15.575373 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 01 22:38:15.575558 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:15.575430 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" podUID="5ec80ece-abb9-45b5-a34e-45bbb2d90eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 01 22:38:16.651351 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:16.651304 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:16.652153 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:16.652133 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:16.653573 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:16.653552 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:38:16.699237 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:16.699190 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:38:16.988522 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:38:16.988474 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:38:17.013970 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:17.013936 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:17.014914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:17.014890 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:17.015556 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:17.015536 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:38:19.463983 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:19.463933 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:19.509988 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:19.509932 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:19.638512 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:19.638471 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded" Nov 01 22:38:19.689973 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:19.689926 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:20.072888 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:20.065962 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:23.015185 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.015138 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr"] Nov 01 22:38:23.020886 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.020861 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:23.040093 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.040056 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr"] Nov 01 22:38:23.068651 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.068617 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq9j6\" (UniqueName: \"kubernetes.io/projected/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-kube-api-access-zq9j6\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:23.068848 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.068676 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-credential-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:23.068915 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.068823 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istio-envoy\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:23.068915 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.068892 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-workload-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:23.069009 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.068929 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istio-token\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:23.069101 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.069084 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-workload-certs\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:23.069134 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.069108 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istio-data\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:23.069134 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.069127 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istiod-ca-cert\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:23.069219 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.069153 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istio-podinfo\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:23.170623 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.170580 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq9j6\" (UniqueName: \"kubernetes.io/projected/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-kube-api-access-zq9j6\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:23.170780 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.170640 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-credential-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:23.170780 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.170684 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istio-envoy\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:23.170780 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.170706 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-workload-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:23.170780 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.170739 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istio-token\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:23.171004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.170824 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-workload-certs\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:23.171004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.170876 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istio-data\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:23.171004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.170896 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istiod-ca-cert\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:23.171004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.170923 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istio-podinfo\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:23.171219 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.171106 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-credential-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:23.171278 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.171209 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-workload-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:23.171315 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.171277 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istio-data\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:23.171372 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.171332 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-workload-certs\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:23.171580 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.171557 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istiod-ca-cert\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:23.173248 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.173223 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istio-envoy\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:23.173482 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.173465 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istio-podinfo\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:23.183651 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.183623 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istio-token\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:23.187400 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.187372 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq9j6\" (UniqueName: \"kubernetes.io/projected/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-kube-api-access-zq9j6\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:23.342441 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.342337 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:23.634227 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.634198 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr"] Nov 01 22:38:23.635819 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:38:23.635794 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e83fcc9_66da_4de1_837a_b9a2a7f7a056.slice/crio-0fb2d04d3e2725391e9be6dcbb5737eb8392f9aec1f05db5d3b918f8907e9270 WatchSource:0}: Error finding container 0fb2d04d3e2725391e9be6dcbb5737eb8392f9aec1f05db5d3b918f8907e9270: Status 404 returned error can't find the container with id 0fb2d04d3e2725391e9be6dcbb5737eb8392f9aec1f05db5d3b918f8907e9270 Nov 01 22:38:23.638283 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.638242 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:38:23.638395 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.638330 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:38:23.638395 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.638376 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:38:23.667899 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.667868 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" event={"ID":"1e83fcc9-66da-4de1-837a-b9a2a7f7a056","Type":"ContainerStarted","Data":"0fb2d04d3e2725391e9be6dcbb5737eb8392f9aec1f05db5d3b918f8907e9270"} Nov 01 22:38:23.782353 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.782314 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv"] Nov 01 22:38:23.787511 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.787489 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:23.811723 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.811700 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv"] Nov 01 22:38:23.882764 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.882718 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbzmv\" (UniqueName: \"kubernetes.io/projected/dccd06ae-f4d0-485f-abe7-89cc6e34a431-kube-api-access-xbzmv\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:23.882983 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.882777 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-workload-certs\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:23.882983 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.882806 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istio-podinfo\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:23.882983 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.882910 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istio-data\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:23.882983 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.882957 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istio-token\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:23.883149 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.883105 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-credential-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:23.883227 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.883211 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istio-envoy\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:23.883336 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.883320 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istiod-ca-cert\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:23.883401 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.883385 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-workload-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:23.943369 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.943330 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw"] Nov 01 22:38:23.947864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.947822 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:23.965750 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.958695 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-f338e4-openshift-default-dockercfg-ksfnw\"" Nov 01 22:38:23.967721 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.967643 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw"] Nov 01 22:38:23.987681 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.987649 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-credential-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:23.987880 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.987725 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istio-envoy\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:23.987880 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.987806 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istiod-ca-cert\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:23.987880 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.987867 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-workload-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:23.988624 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.988576 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbzmv\" (UniqueName: \"kubernetes.io/projected/dccd06ae-f4d0-485f-abe7-89cc6e34a431-kube-api-access-xbzmv\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:23.988745 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.988652 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-workload-certs\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:23.988745 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.988678 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istio-podinfo\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:23.988745 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.988707 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istio-data\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:23.988745 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.988740 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istio-token\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:23.989786 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.989374 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-credential-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:23.990408 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.990385 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istiod-ca-cert\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:23.990697 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.990678 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-workload-certs\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:23.990774 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.990754 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-workload-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:23.990964 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.990945 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istio-data\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:23.994983 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.994961 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istio-envoy\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:23.996436 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.996353 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istio-podinfo\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:23.999156 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.999137 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istio-token\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:23.999601 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:23.999578 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbzmv\" (UniqueName: \"kubernetes.io/projected/dccd06ae-f4d0-485f-abe7-89cc6e34a431-kube-api-access-xbzmv\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:24.089677 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.089642 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5913d691-0d87-4599-b176-d132888e187e-istio-token\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:24.090173 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.089765 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-istio-data\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:24.090173 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.089817 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4xtk\" (UniqueName: \"kubernetes.io/projected/5913d691-0d87-4599-b176-d132888e187e-kube-api-access-t4xtk\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:24.090173 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.089974 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5913d691-0d87-4599-b176-d132888e187e-istiod-ca-cert\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:24.090173 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.090035 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-istio-envoy\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:24.090173 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.090096 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-workload-certs\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:24.090173 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.090124 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-credential-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:24.090485 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.090214 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5913d691-0d87-4599-b176-d132888e187e-istio-podinfo\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:24.090485 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.090282 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-workload-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:24.114191 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.114156 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:24.193420 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.191216 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-credential-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:24.193420 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.191334 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5913d691-0d87-4599-b176-d132888e187e-istio-podinfo\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:24.193420 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.191424 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-workload-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:24.193420 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.191519 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5913d691-0d87-4599-b176-d132888e187e-istio-token\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:24.193420 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.191573 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-istio-data\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:24.193420 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.191631 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4xtk\" (UniqueName: \"kubernetes.io/projected/5913d691-0d87-4599-b176-d132888e187e-kube-api-access-t4xtk\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:24.193420 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.191681 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5913d691-0d87-4599-b176-d132888e187e-istiod-ca-cert\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:24.193420 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.191738 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-istio-envoy\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:24.193420 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.191803 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-workload-certs\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:24.193420 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.192275 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-workload-certs\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:24.193420 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.192529 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-credential-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:24.193420 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.193272 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-istio-data\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:24.194138 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.193536 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-workload-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:24.195088 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.194475 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5913d691-0d87-4599-b176-d132888e187e-istiod-ca-cert\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:24.203161 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.203134 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4xtk\" (UniqueName: \"kubernetes.io/projected/5913d691-0d87-4599-b176-d132888e187e-kube-api-access-t4xtk\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:24.210384 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.210359 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-istio-envoy\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:24.210722 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.210694 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5913d691-0d87-4599-b176-d132888e187e-istio-podinfo\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:24.212991 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.212968 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5913d691-0d87-4599-b176-d132888e187e-istio-token\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:24.272010 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.271970 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:24.470910 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.467791 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv"] Nov 01 22:38:24.495129 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.495085 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:38:24.495254 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.495172 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:38:24.495254 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.495219 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:38:24.578512 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.578484 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw"] Nov 01 22:38:24.580195 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:38:24.580166 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5913d691_0d87_4599_b176_d132888e187e.slice/crio-0753b2bac246857d4110979814c7df068dd1ffa3d21e51fc37b28ea4878a4383 WatchSource:0}: Error finding container 0753b2bac246857d4110979814c7df068dd1ffa3d21e51fc37b28ea4878a4383: Status 404 returned error can't find the container with id 0753b2bac246857d4110979814c7df068dd1ffa3d21e51fc37b28ea4878a4383 Nov 01 22:38:24.582951 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.582919 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:38:24.583119 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.583101 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:38:24.583215 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.583202 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:38:24.683432 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.683389 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" event={"ID":"5913d691-0d87-4599-b176-d132888e187e","Type":"ContainerStarted","Data":"0753b2bac246857d4110979814c7df068dd1ffa3d21e51fc37b28ea4878a4383"} Nov 01 22:38:24.686627 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.685506 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" event={"ID":"1e83fcc9-66da-4de1-837a-b9a2a7f7a056","Type":"ContainerStarted","Data":"8c9bf38e318c286e9bc5fede40f45d98baea06953a0d1953a546e8be1306104e"} Nov 01 22:38:24.689469 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.688959 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" event={"ID":"dccd06ae-f4d0-485f-abe7-89cc6e34a431","Type":"ContainerStarted","Data":"820d6e0754c71df0d52b9e881cd5f7f91453d3c741158ca06f58fa76b74fbcbd"} Nov 01 22:38:24.708596 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.708532 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" podStartSLOduration=2.708512519 podStartE2EDuration="2.708512519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:24.707695072 +0000 UTC m=+873.112285704" watchObservedRunningTime="2026-11-01 22:38:24.708512519 +0000 UTC m=+873.113103152" Nov 01 22:38:24.759917 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:24.758223 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" podStartSLOduration=1.758199824 podStartE2EDuration="1.758199824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:24.730030048 +0000 UTC m=+873.134620681" watchObservedRunningTime="2026-11-01 22:38:24.758199824 +0000 UTC m=+873.162790460" Nov 01 22:38:25.114745 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:25.114671 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:25.115971 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:25.115947 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 22:38:25.116084 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:25.115994 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" podUID="dccd06ae-f4d0-485f-abe7-89cc6e34a431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 22:38:25.348857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:25.344189 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:25.360861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:25.360005 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 01 22:38:25.360861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:25.360081 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" podUID="1e83fcc9-66da-4de1-837a-b9a2a7f7a056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 01 22:38:25.707914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:25.707875 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" event={"ID":"5913d691-0d87-4599-b176-d132888e187e","Type":"ContainerStarted","Data":"b4167522c4317755f974ae58b3e915c0f4775a3c9bce3630c0b5d6b0546e2fb4"} Nov 01 22:38:25.719187 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:25.713265 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" event={"ID":"dccd06ae-f4d0-485f-abe7-89cc6e34a431","Type":"ContainerStarted","Data":"d6228b94f0a9bed0f8a9a0ad888a3634ffa204a3ee67fd44fe038363d574f9ca"} Nov 01 22:38:26.018246 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:26.017698 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" podStartSLOduration=3.017674891 podStartE2EDuration="3.017674891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:26.015858756 +0000 UTC m=+874.420449409" watchObservedRunningTime="2026-11-01 22:38:26.017674891 +0000 UTC m=+874.422265524" Nov 01 22:38:26.115853 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:26.115262 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 22:38:26.115853 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:26.115324 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" podUID="dccd06ae-f4d0-485f-abe7-89cc6e34a431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 22:38:26.350924 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:26.344207 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 01 22:38:26.350924 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:26.344266 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" podUID="1e83fcc9-66da-4de1-837a-b9a2a7f7a056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 01 22:38:26.419858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:26.418465 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:26.482858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:26.481976 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 01 22:38:26.482858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:26.482043 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" podUID="5913d691-0d87-4599-b176-d132888e187e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 01 22:38:27.120896 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:27.119640 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 22:38:27.120896 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:27.119706 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" podUID="dccd06ae-f4d0-485f-abe7-89cc6e34a431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 22:38:27.350716 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:27.350671 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 01 22:38:27.350946 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:27.350737 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" podUID="1e83fcc9-66da-4de1-837a-b9a2a7f7a056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 01 22:38:27.372941 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:27.353162 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 01 22:38:27.372941 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:27.353220 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" podUID="5913d691-0d87-4599-b176-d132888e187e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 01 22:38:28.117441 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:28.117401 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 22:38:28.117770 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:28.117725 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" podUID="dccd06ae-f4d0-485f-abe7-89cc6e34a431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 22:38:28.310861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:28.310531 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 01 22:38:28.310861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:28.310603 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" podUID="5913d691-0d87-4599-b176-d132888e187e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 01 22:38:28.345859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:28.345324 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 01 22:38:28.345859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:28.345408 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" podUID="1e83fcc9-66da-4de1-837a-b9a2a7f7a056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 01 22:38:29.123262 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:29.120243 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 22:38:29.123262 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:29.120305 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" podUID="dccd06ae-f4d0-485f-abe7-89cc6e34a431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 22:38:29.273858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:29.273271 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 01 22:38:29.273858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:29.273333 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" podUID="5913d691-0d87-4599-b176-d132888e187e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 01 22:38:29.349697 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:29.349217 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 01 22:38:29.349697 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:29.349281 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" podUID="1e83fcc9-66da-4de1-837a-b9a2a7f7a056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 01 22:38:29.697299 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:29.697266 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn"] Nov 01 22:38:29.738182 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:29.738145 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn"] Nov 01 22:38:29.739442 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:29.738275 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:38:29.798035 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:29.776037 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9ad303\"/\"distinct-2-w2-9ad303-kserve-self-signed-certs\"" Nov 01 22:38:29.911876 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:29.877948 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f3b7d0-5402-4055-82cf-f124eb0a2af6-tls-certs\") pod \"distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn\" (UID: \"37f3b7d0-5402-4055-82cf-f124eb0a2af6\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:38:29.911876 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:29.878005 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlgq7\" (UniqueName: \"kubernetes.io/projected/37f3b7d0-5402-4055-82cf-f124eb0a2af6-kube-api-access-qlgq7\") pod \"distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn\" (UID: \"37f3b7d0-5402-4055-82cf-f124eb0a2af6\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:38:29.941865 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:29.931090 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f3b7d0-5402-4055-82cf-f124eb0a2af6-tmp-dir\") pod \"distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn\" (UID: \"37f3b7d0-5402-4055-82cf-f124eb0a2af6\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:38:29.941865 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:29.931150 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37f3b7d0-5402-4055-82cf-f124eb0a2af6-home\") pod \"distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn\" (UID: \"37f3b7d0-5402-4055-82cf-f124eb0a2af6\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:38:29.941865 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:29.931244 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37f3b7d0-5402-4055-82cf-f124eb0a2af6-dshm\") pod \"distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn\" (UID: \"37f3b7d0-5402-4055-82cf-f124eb0a2af6\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:38:29.941865 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:29.931373 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37f3b7d0-5402-4055-82cf-f124eb0a2af6-model-cache\") pod \"distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn\" (UID: \"37f3b7d0-5402-4055-82cf-f124eb0a2af6\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:38:30.040809 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:30.040770 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37f3b7d0-5402-4055-82cf-f124eb0a2af6-model-cache\") pod \"distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn\" (UID: \"37f3b7d0-5402-4055-82cf-f124eb0a2af6\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:38:30.041114 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:30.041091 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f3b7d0-5402-4055-82cf-f124eb0a2af6-tls-certs\") pod \"distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn\" (UID: \"37f3b7d0-5402-4055-82cf-f124eb0a2af6\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:38:30.041251 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:30.041234 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlgq7\" (UniqueName: \"kubernetes.io/projected/37f3b7d0-5402-4055-82cf-f124eb0a2af6-kube-api-access-qlgq7\") pod \"distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn\" (UID: \"37f3b7d0-5402-4055-82cf-f124eb0a2af6\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:38:30.041490 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:30.041473 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f3b7d0-5402-4055-82cf-f124eb0a2af6-tmp-dir\") pod \"distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn\" (UID: \"37f3b7d0-5402-4055-82cf-f124eb0a2af6\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:38:30.041608 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:30.041594 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37f3b7d0-5402-4055-82cf-f124eb0a2af6-home\") pod \"distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn\" (UID: \"37f3b7d0-5402-4055-82cf-f124eb0a2af6\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:38:30.041790 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:30.041773 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37f3b7d0-5402-4055-82cf-f124eb0a2af6-dshm\") pod \"distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn\" (UID: \"37f3b7d0-5402-4055-82cf-f124eb0a2af6\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:38:30.043554 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:30.043528 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37f3b7d0-5402-4055-82cf-f124eb0a2af6-model-cache\") pod \"distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn\" (UID: \"37f3b7d0-5402-4055-82cf-f124eb0a2af6\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:38:30.044513 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:30.044414 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f3b7d0-5402-4055-82cf-f124eb0a2af6-tmp-dir\") pod \"distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn\" (UID: \"37f3b7d0-5402-4055-82cf-f124eb0a2af6\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:38:30.089855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:30.078793 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37f3b7d0-5402-4055-82cf-f124eb0a2af6-home\") pod \"distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn\" (UID: \"37f3b7d0-5402-4055-82cf-f124eb0a2af6\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:38:30.096275 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:30.096240 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37f3b7d0-5402-4055-82cf-f124eb0a2af6-dshm\") pod \"distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn\" (UID: \"37f3b7d0-5402-4055-82cf-f124eb0a2af6\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:38:30.113884 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:30.111509 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f3b7d0-5402-4055-82cf-f124eb0a2af6-tls-certs\") pod \"distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn\" (UID: \"37f3b7d0-5402-4055-82cf-f124eb0a2af6\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:38:30.115215 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:30.115183 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 22:38:30.115453 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:30.115424 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" podUID="dccd06ae-f4d0-485f-abe7-89cc6e34a431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 22:38:30.127417 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:30.127346 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlgq7\" (UniqueName: \"kubernetes.io/projected/37f3b7d0-5402-4055-82cf-f124eb0a2af6-kube-api-access-qlgq7\") pod \"distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn\" (UID: \"37f3b7d0-5402-4055-82cf-f124eb0a2af6\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:38:30.292920 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:30.291936 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 01 22:38:30.292920 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:30.291995 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" podUID="5913d691-0d87-4599-b176-d132888e187e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 01 22:38:30.344861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:30.343225 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 01 22:38:30.344861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:30.343282 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" podUID="1e83fcc9-66da-4de1-837a-b9a2a7f7a056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 01 22:38:30.366859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:30.364629 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:38:30.810614 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:30.810548 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn"] Nov 01 22:38:31.009427 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:31.009380 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" event={"ID":"37f3b7d0-5402-4055-82cf-f124eb0a2af6","Type":"ContainerStarted","Data":"2d901a2b05894e46dbfa113a1d03a8c2dd3f2926b2d0bcbcbfbc821682db5071"} Nov 01 22:38:31.009611 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:31.009438 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" event={"ID":"37f3b7d0-5402-4055-82cf-f124eb0a2af6","Type":"ContainerStarted","Data":"e76c0f1187e465554360d3f9331910c47554d9454589ed4e0d0676c8619701c6"} Nov 01 22:38:31.028569 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:31.028500 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podStartSLOduration=2.028482028 podStartE2EDuration="2.028482028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:31.027513898 +0000 UTC m=+879.432104531" watchObservedRunningTime="2026-11-01 22:38:31.028482028 +0000 UTC m=+879.433072661" Nov 01 22:38:31.115433 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:31.115344 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 22:38:31.115594 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:31.115433 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" podUID="dccd06ae-f4d0-485f-abe7-89cc6e34a431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 22:38:31.287171 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:31.287132 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 01 22:38:31.287461 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:31.287430 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" podUID="5913d691-0d87-4599-b176-d132888e187e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 01 22:38:31.351293 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:31.351255 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 01 22:38:31.351590 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:31.351557 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" podUID="1e83fcc9-66da-4de1-837a-b9a2a7f7a056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 01 22:38:31.365522 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:31.365436 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:38:32.118033 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:32.116442 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 22:38:32.118033 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:32.116505 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" podUID="dccd06ae-f4d0-485f-abe7-89cc6e34a431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 22:38:32.272470 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:32.272439 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 01 22:38:32.272664 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:32.272488 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" podUID="5913d691-0d87-4599-b176-d132888e187e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 01 22:38:32.349040 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:32.349005 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:32.349649 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:32.349628 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:32.350465 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:32.350444 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:38:33.119599 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:33.119561 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:33.120418 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:33.120397 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:33.121145 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:33.121123 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:38:33.283806 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:33.283773 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:33.286686 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:33.284803 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:33.286951 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:33.286912 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 22:38:37.933973 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:37.933936 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh"] Nov 01 22:38:37.938234 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:37.938214 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:37.951499 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:37.951473 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg"] Nov 01 22:38:37.956120 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:37.956095 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:37.957716 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:37.957689 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh"] Nov 01 22:38:37.968066 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:37.967641 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istio-data\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:37.968066 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:37.967680 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5fsb\" (UniqueName: \"kubernetes.io/projected/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-kube-api-access-v5fsb\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:37.968066 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:37.967716 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-workload-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:37.968066 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:37.967873 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324f88f5-9089-49ad-9a07-22abe2a0e98f-istiod-ca-cert\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:37.968066 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:37.967920 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-workload-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:37.968066 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:37.968029 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istio-envoy\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:37.968422 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:37.968075 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324f88f5-9089-49ad-9a07-22abe2a0e98f-istio-token\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:37.968422 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:37.968125 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-istio-envoy\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:37.968422 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:37.968237 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324f88f5-9089-49ad-9a07-22abe2a0e98f-istio-podinfo\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:37.968422 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:37.968280 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-credential-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:37.968422 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:37.968381 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istio-podinfo\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:37.968422 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:37.968411 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istio-token\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:37.968720 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:37.968511 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-istio-data\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:37.968720 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:37.968548 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istiod-ca-cert\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:37.968720 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:37.968672 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-credential-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:37.968720 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:37.968707 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-workload-certs\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:37.969013 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:37.968740 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-workload-certs\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:37.969013 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:37.968891 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfjfv\" (UniqueName: \"kubernetes.io/projected/324f88f5-9089-49ad-9a07-22abe2a0e98f-kube-api-access-hfjfv\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:37.977209 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:37.977177 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg"] Nov 01 22:38:38.069890 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.069823 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istio-podinfo\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:38.070089 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.069913 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istio-token\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:38.070089 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.069968 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-istio-data\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:38.070089 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.069997 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istiod-ca-cert\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:38.070281 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.070093 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-credential-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:38.070281 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.070118 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-workload-certs\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:38.070281 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.070148 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-workload-certs\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:38.070281 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.070212 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfjfv\" (UniqueName: \"kubernetes.io/projected/324f88f5-9089-49ad-9a07-22abe2a0e98f-kube-api-access-hfjfv\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:38.070281 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.070271 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istio-data\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:38.070557 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.070296 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5fsb\" (UniqueName: \"kubernetes.io/projected/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-kube-api-access-v5fsb\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:38.070557 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.070336 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-workload-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:38.070557 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.070356 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-istio-data\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:38.070557 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.070384 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324f88f5-9089-49ad-9a07-22abe2a0e98f-istiod-ca-cert\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:38.070557 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.070437 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-workload-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:38.070557 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.070522 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istio-envoy\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:38.070557 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.070532 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-credential-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:38.070968 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.070595 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324f88f5-9089-49ad-9a07-22abe2a0e98f-istio-token\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:38.070968 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.070634 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-istio-envoy\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:38.070968 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.070697 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324f88f5-9089-49ad-9a07-22abe2a0e98f-istio-podinfo\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:38.070968 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.070742 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-credential-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:38.070968 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.070856 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-workload-certs\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:38.070968 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.070891 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istiod-ca-cert\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:38.071298 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.071048 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324f88f5-9089-49ad-9a07-22abe2a0e98f-istiod-ca-cert\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:38.071298 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.071137 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-credential-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:38.071298 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.071176 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-workload-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:38.071478 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.071383 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-workload-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:38.071587 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.071549 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istio-data\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:38.071843 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.071742 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-workload-certs\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:38.073257 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.073227 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-istio-envoy\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:38.073388 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.073330 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istio-envoy\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:38.073388 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.073354 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istio-podinfo\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:38.073512 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.073498 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324f88f5-9089-49ad-9a07-22abe2a0e98f-istio-podinfo\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:38.092156 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.092125 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfjfv\" (UniqueName: \"kubernetes.io/projected/324f88f5-9089-49ad-9a07-22abe2a0e98f-kube-api-access-hfjfv\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:38.092314 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.092169 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istio-token\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:38.092455 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.092436 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5fsb\" (UniqueName: \"kubernetes.io/projected/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-kube-api-access-v5fsb\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:38.093264 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.093240 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324f88f5-9089-49ad-9a07-22abe2a0e98f-istio-token\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:38.256091 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.256004 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:38.277387 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.277350 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:38.606154 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.605856 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh"] Nov 01 22:38:38.611103 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.611016 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:38:38.611175 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.611101 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:38:38.611175 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.611140 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:38:38.612713 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.612690 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg"] Nov 01 22:38:38.614507 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:38:38.614482 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a107568_bb23_4c0f_ad9e_cdfb206ce4bf.slice/crio-8637bcb6db822b3961f2828588559b09ab1d68f8be4a78f7e47c5f072d29b444 WatchSource:0}: Error finding container 8637bcb6db822b3961f2828588559b09ab1d68f8be4a78f7e47c5f072d29b444: Status 404 returned error can't find the container with id 8637bcb6db822b3961f2828588559b09ab1d68f8be4a78f7e47c5f072d29b444 Nov 01 22:38:38.620590 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.620554 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:38:38.620679 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.620645 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:38:38.620721 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.620690 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:38:38.977706 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.977672 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr"] Nov 01 22:38:38.982941 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.982915 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:38.994336 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:38.994307 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr"] Nov 01 22:38:39.087705 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.087667 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dfl9\" (UniqueName: \"kubernetes.io/projected/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-kube-api-access-8dfl9\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:39.087914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.087737 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-credential-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:39.087914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.087854 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istio-podinfo\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:39.088062 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.087944 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istio-data\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:39.088062 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.087978 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-workload-certs\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:39.088062 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.088005 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istiod-ca-cert\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:39.088062 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.088049 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-workload-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:39.088277 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.088077 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istio-token\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:39.088277 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.088264 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istio-envoy\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:39.104493 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.104464 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" event={"ID":"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf","Type":"ContainerStarted","Data":"46c43d91908ed3a943a1ac875128656ebe9a2bb373a47f725e9af396f7b1c602"} Nov 01 22:38:39.104653 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.104517 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" event={"ID":"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf","Type":"ContainerStarted","Data":"8637bcb6db822b3961f2828588559b09ab1d68f8be4a78f7e47c5f072d29b444"} Nov 01 22:38:39.106351 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.106318 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" event={"ID":"324f88f5-9089-49ad-9a07-22abe2a0e98f","Type":"ContainerStarted","Data":"121915229adc2402e6e71be94af2d71d0587e69aae763c9d54e5088bbc4c6e84"} Nov 01 22:38:39.106454 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.106372 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" event={"ID":"324f88f5-9089-49ad-9a07-22abe2a0e98f","Type":"ContainerStarted","Data":"b089f9aa7c6260b405c64bd1afc25bb3e53a7bf414b58e92587e16b63a3df0d3"} Nov 01 22:38:39.126264 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.126193 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" podStartSLOduration=2.126180475 podStartE2EDuration="2.126180475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:39.125184935 +0000 UTC m=+887.529775568" watchObservedRunningTime="2026-11-01 22:38:39.126180475 +0000 UTC m=+887.530771098" Nov 01 22:38:39.147718 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.147670 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" podStartSLOduration=2.147656315 podStartE2EDuration="2.147656315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:39.145007509 +0000 UTC m=+887.549598153" watchObservedRunningTime="2026-11-01 22:38:39.147656315 +0000 UTC m=+887.552246947" Nov 01 22:38:39.190957 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.190916 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-credential-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:39.191167 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.191074 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istio-podinfo\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:39.191379 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.191346 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-credential-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:39.191488 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.191468 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istio-data\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:39.191553 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.191528 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-workload-certs\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:39.191612 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.191574 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istio-data\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:39.191665 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.191638 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istiod-ca-cert\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:39.191881 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.191824 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-workload-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:39.192039 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.192024 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istio-token\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:39.192789 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.192124 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-workload-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:39.192789 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.192265 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istiod-ca-cert\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:39.192789 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.192297 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istio-envoy\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:39.192789 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.192409 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dfl9\" (UniqueName: \"kubernetes.io/projected/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-kube-api-access-8dfl9\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:39.193043 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.192928 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-workload-certs\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:39.194717 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.194697 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istio-podinfo\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:39.196105 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.196084 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istio-envoy\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:39.200561 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.200536 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istio-token\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:39.200644 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.200590 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dfl9\" (UniqueName: \"kubernetes.io/projected/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-kube-api-access-8dfl9\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:39.257229 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.257142 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:39.258516 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.258492 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 01 22:38:39.258648 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.258538 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" podUID="324f88f5-9089-49ad-9a07-22abe2a0e98f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 01 22:38:39.277917 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.277888 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:39.279688 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.279659 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 22:38:39.279864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.279713 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" podUID="5a107568-bb23-4c0f-ad9e-cdfb206ce4bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 22:38:39.305214 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.302238 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:39.923933 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.923898 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr"] Nov 01 22:38:39.925999 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:38:39.925960 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83c24a9_8ffb_4b1b_910a_c7c9b0fadef6.slice/crio-798cd7c221dc74648db6dc509de988af50dadc8670dca0da30eb065fde095982 WatchSource:0}: Error finding container 798cd7c221dc74648db6dc509de988af50dadc8670dca0da30eb065fde095982: Status 404 returned error can't find the container with id 798cd7c221dc74648db6dc509de988af50dadc8670dca0da30eb065fde095982 Nov 01 22:38:39.928603 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.928568 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:38:39.928703 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.928630 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:38:39.928703 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:39.928657 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:38:40.116372 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:40.116335 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" event={"ID":"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6","Type":"ContainerStarted","Data":"9d72fc67ed44c15059a2173bde04077e4daadd84aa24f1fe9dd1fb06154f7f8a"} Nov 01 22:38:40.116372 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:40.116381 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" event={"ID":"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6","Type":"ContainerStarted","Data":"798cd7c221dc74648db6dc509de988af50dadc8670dca0da30eb065fde095982"} Nov 01 22:38:40.158036 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:40.157033 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" podStartSLOduration=2.157008163 podStartE2EDuration="2.157008163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:40.154693742 +0000 UTC m=+888.559284373" watchObservedRunningTime="2026-11-01 22:38:40.157008163 +0000 UTC m=+888.561598793" Nov 01 22:38:40.273932 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:40.261241 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 01 22:38:40.273932 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:40.261307 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" podUID="324f88f5-9089-49ad-9a07-22abe2a0e98f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 01 22:38:40.281861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:40.279217 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 22:38:40.281861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:40.279275 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" podUID="5a107568-bb23-4c0f-ad9e-cdfb206ce4bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 22:38:40.312877 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:40.303406 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:40.380001 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:40.379064 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:38:40.610175 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:40.610065 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:38:41.221484 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:41.219285 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:38:41.263857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:41.259722 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 01 22:38:41.263857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:41.259784 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" podUID="324f88f5-9089-49ad-9a07-22abe2a0e98f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 01 22:38:41.296859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:41.281202 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 22:38:41.297188 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:41.297150 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" podUID="5a107568-bb23-4c0f-ad9e-cdfb206ce4bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 22:38:41.330867 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:41.330267 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:38:41.330867 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:41.330359 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" podUID="a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 01 22:38:42.278043 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:42.277586 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 01 22:38:42.278043 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:42.277647 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" podUID="324f88f5-9089-49ad-9a07-22abe2a0e98f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 01 22:38:42.279089 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:42.278997 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 22:38:42.279089 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:42.279042 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" podUID="5a107568-bb23-4c0f-ad9e-cdfb206ce4bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 22:38:42.320320 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:42.320282 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:38:42.320606 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:42.320577 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" podUID="a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 01 22:38:43.264284 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:43.264245 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 01 22:38:43.265494 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:43.265458 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" podUID="324f88f5-9089-49ad-9a07-22abe2a0e98f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 01 22:38:43.288859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:43.278189 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 22:38:43.288859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:43.278246 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" podUID="5a107568-bb23-4c0f-ad9e-cdfb206ce4bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 22:38:43.310858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:43.310162 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:38:43.310858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:43.310236 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" podUID="a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 01 22:38:44.259877 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:44.257346 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 01 22:38:44.259877 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:44.257405 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" podUID="324f88f5-9089-49ad-9a07-22abe2a0e98f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 01 22:38:44.281388 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:44.281354 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 22:38:44.281653 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:44.281623 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" podUID="5a107568-bb23-4c0f-ad9e-cdfb206ce4bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 22:38:44.304023 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:44.303990 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:38:44.304606 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:44.304573 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" podUID="a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 01 22:38:45.266857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:45.266404 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 01 22:38:45.266857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:45.266463 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" podUID="324f88f5-9089-49ad-9a07-22abe2a0e98f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 01 22:38:45.338565 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:45.338529 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:38:45.339170 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:45.339134 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" podUID="a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 01 22:38:45.342911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:45.339344 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 22:38:45.346952 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:45.346915 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" podUID="5a107568-bb23-4c0f-ad9e-cdfb206ce4bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 22:38:46.176858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:46.176019 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:46.259266 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:46.259219 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 01 22:38:46.259551 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:46.259507 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" podUID="324f88f5-9089-49ad-9a07-22abe2a0e98f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 01 22:38:46.281343 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:46.281308 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 22:38:46.281629 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:46.281597 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" podUID="5a107568-bb23-4c0f-ad9e-cdfb206ce4bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 22:38:46.305590 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:46.305554 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:38:46.305869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:46.305822 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" podUID="a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 01 22:38:46.313261 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:46.313223 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:46.427043 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:46.426944 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 01 22:38:46.459657 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:46.457310 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:46.459657 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:46.457695 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:38:46.459657 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:46.459385 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:46.522068 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:46.522005 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:46.694775 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:46.694653 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:46.741111 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:46.741056 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:47.036195 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:47.035113 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:47.119471 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:38:47.119399 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:38:47.179158 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:47.176924 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:47.179158 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:47.177563 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:38:47.262037 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:47.261566 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 01 22:38:47.262037 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:47.261626 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" podUID="324f88f5-9089-49ad-9a07-22abe2a0e98f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 01 22:38:47.284848 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:47.284711 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 22:38:47.284848 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:47.284767 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" podUID="5a107568-bb23-4c0f-ad9e-cdfb206ce4bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 22:38:47.308675 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:47.308577 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:38:47.308994 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:47.308892 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" podUID="a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 01 22:38:47.436863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:47.427982 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:47.436863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:47.428691 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:38:47.466046 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:47.466006 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:38:47.466647 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:47.466617 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:38:47.467068 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:47.467043 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:38:47.467810 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:47.467790 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:38:47.531021 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:47.530972 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:47.656698 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:47.634814 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:47.701874 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:47.697148 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:47.754859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:47.739183 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:38:47.968859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:47.965178 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:48.051388 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:48.049568 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:48.097862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:48.085035 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:38:48.097862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:48.085126 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:38:48.097862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:48.085185 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:38:48.097862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:48.085236 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:38:48.097862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:48.085287 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:38:48.161857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:48.161315 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:48.187857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:48.187265 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 01 22:38:48.369915 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:48.368958 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:38:48.369915 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:48.369025 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" podUID="a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 01 22:38:48.400725 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:48.399935 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:38:48.400725 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:48.400001 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" podUID="1e83fcc9-66da-4de1-837a-b9a2a7f7a056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:48.436859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:48.429930 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:48.471854 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:48.467926 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:48.514190 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:48.487381 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:48.562856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:48.487417 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:48.816888 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:48.749187 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:48.816888 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:48.749685 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 22:38:48.816888 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:48.752612 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:38:49.110136 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:49.110028 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:38:49.177009 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:49.176969 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:38:49.177330 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:49.177299 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" podUID="dccd06ae-f4d0-485f-abe7-89cc6e34a431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:49.196861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:49.195219 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 01 22:38:49.316967 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:49.314147 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:38:49.316967 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:49.314204 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" podUID="a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 01 22:38:49.317391 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:49.317351 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:38:49.317607 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:49.317565 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" podUID="324f88f5-9089-49ad-9a07-22abe2a0e98f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:49.357023 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:49.356973 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:38:49.357306 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:49.357259 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" podUID="5a107568-bb23-4c0f-ad9e-cdfb206ce4bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:49.436871 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:49.431671 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:49.514941 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:49.514888 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:38:49.515926 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:49.515894 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:38:49.564031 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:49.563973 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:38:49.678356 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:49.677953 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:38:49.741014 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:49.740913 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded" Nov 01 22:38:49.761865 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:49.755000 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:49.762519 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:49.755034 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 01 22:38:49.766925 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:49.755052 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 22:38:49.809681 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:49.809641 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:38:50.196182 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:50.196134 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 01 22:38:50.197817 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:50.197784 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:50.240861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:50.210457 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:50.266617 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:50.266555 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:50.316754 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:50.316715 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:38:50.317643 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:50.317616 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:38:50.333893 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:50.317801 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" podUID="a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 01 22:38:50.432423 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:50.432342 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:50.464023 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:50.453908 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:38:50.464023 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:50.453985 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" podUID="5a107568-bb23-4c0f-ad9e-cdfb206ce4bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:50.536235 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:50.536061 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:38:50.540196 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:50.538057 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:38:50.683241 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:50.683206 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:38:50.793913 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:50.788933 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:38:50.793913 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:50.771288 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:50.793913 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:50.771325 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:38:50.840886 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:50.834426 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:38:50.840886 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:50.834495 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" podUID="db213002-5fde-4c7a-ba7e-6762c011b4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:50.908859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:50.906893 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:38:51.248861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:51.248809 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:38:51.252139 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:51.252117 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:38:51.388272 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:51.387692 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:38:51.458682 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:51.458648 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:38:51.458997 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:51.458968 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" podUID="5a107568-bb23-4c0f-ad9e-cdfb206ce4bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:51.468136 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:51.468105 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:38:52.137052 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:52.137011 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:52.138649 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:52.138619 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:52.163182 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:52.163150 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:38:52.483719 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:52.483681 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:38:52.484033 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:52.484003 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" podUID="5a107568-bb23-4c0f-ad9e-cdfb206ce4bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:52.504300 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:52.504246 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:52.506142 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:52.506119 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:52.547856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:52.547422 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:38:53.131545 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.131509 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7"] Nov 01 22:38:53.155384 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.154410 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7"] Nov 01 22:38:53.155384 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.154528 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:38:53.228009 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.227976 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81b97a5-fd37-4296-8c16-94f46421d4a3-istio-token\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:38:53.228169 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.228106 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-istio-envoy\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:38:53.228237 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.228198 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81b97a5-fd37-4296-8c16-94f46421d4a3-istiod-ca-cert\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:38:53.228281 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.228252 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81b97a5-fd37-4296-8c16-94f46421d4a3-istio-podinfo\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:38:53.228410 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.228379 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-workload-certs\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:38:53.228544 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.228520 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-istio-data\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:38:53.228603 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.228548 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-workload-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:38:53.228603 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.228580 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-credential-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:38:53.228706 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.228664 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b52t2\" (UniqueName: \"kubernetes.io/projected/d81b97a5-fd37-4296-8c16-94f46421d4a3-kube-api-access-b52t2\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:38:53.330903 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.330857 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-istio-envoy\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:38:53.331192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.331168 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81b97a5-fd37-4296-8c16-94f46421d4a3-istiod-ca-cert\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:38:53.331378 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.331361 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81b97a5-fd37-4296-8c16-94f46421d4a3-istio-podinfo\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:38:53.331554 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.331538 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-workload-certs\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:38:53.335870 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.331865 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-istio-data\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:38:53.355681 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.355637 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-workload-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:38:53.359863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.357952 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-credential-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:38:53.359863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.358126 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b52t2\" (UniqueName: \"kubernetes.io/projected/d81b97a5-fd37-4296-8c16-94f46421d4a3-kube-api-access-b52t2\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:38:53.359863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.358332 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81b97a5-fd37-4296-8c16-94f46421d4a3-istio-token\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:38:53.359863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.359054 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-workload-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:38:53.359863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.359670 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-credential-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:38:53.364799 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.360929 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-workload-certs\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:38:53.384644 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.380715 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81b97a5-fd37-4296-8c16-94f46421d4a3-istiod-ca-cert\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:38:53.389252 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.389194 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-istio-data\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:38:53.390259 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.390226 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81b97a5-fd37-4296-8c16-94f46421d4a3-istio-token\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:38:53.392058 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.391848 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-istio-envoy\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:38:53.392058 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.391906 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b52t2\" (UniqueName: \"kubernetes.io/projected/d81b97a5-fd37-4296-8c16-94f46421d4a3-kube-api-access-b52t2\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:38:53.392283 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.392090 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81b97a5-fd37-4296-8c16-94f46421d4a3-istio-podinfo\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:38:53.484609 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.484573 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:38:53.791033 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.790999 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7"] Nov 01 22:38:53.791403 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:38:53.791361 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd81b97a5_fd37_4296_8c16_94f46421d4a3.slice/crio-ea6e44efc9c1b970fe985b2d377b6c5b480e58d3566bcdaf838cac24b4b89130 WatchSource:0}: Error finding container ea6e44efc9c1b970fe985b2d377b6c5b480e58d3566bcdaf838cac24b4b89130: Status 404 returned error can't find the container with id ea6e44efc9c1b970fe985b2d377b6c5b480e58d3566bcdaf838cac24b4b89130 Nov 01 22:38:53.794624 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.794587 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:38:53.794712 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.794655 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:38:53.794712 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:53.794687 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:38:54.256914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:54.256006 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" event={"ID":"d81b97a5-fd37-4296-8c16-94f46421d4a3","Type":"ContainerStarted","Data":"559c98bafa33e8f3b1e8e3c6da6b70b260fff2325348be1dfb82919ee170e312"} Nov 01 22:38:54.256914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:54.256077 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" event={"ID":"d81b97a5-fd37-4296-8c16-94f46421d4a3","Type":"ContainerStarted","Data":"ea6e44efc9c1b970fe985b2d377b6c5b480e58d3566bcdaf838cac24b4b89130"} Nov 01 22:38:54.279918 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:54.279854 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" podStartSLOduration=1.2798145459999999 podStartE2EDuration="1.279814546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:54.277690695 +0000 UTC m=+902.682281328" watchObservedRunningTime="2026-11-01 22:38:54.279814546 +0000 UTC m=+902.684405178" Nov 01 22:38:54.485009 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:54.484975 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:38:54.486713 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:54.486686 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:38:54.486850 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:54.486736 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" podUID="d81b97a5-fd37-4296-8c16-94f46421d4a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 01 22:38:55.485649 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:55.485613 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:38:55.486065 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:55.485668 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" podUID="d81b97a5-fd37-4296-8c16-94f46421d4a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 01 22:38:56.485322 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:56.485280 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:38:56.485507 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:56.485343 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" podUID="d81b97a5-fd37-4296-8c16-94f46421d4a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 01 22:38:57.485409 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:57.485369 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:38:57.485923 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:57.485431 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" podUID="d81b97a5-fd37-4296-8c16-94f46421d4a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 01 22:38:58.485943 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:58.485909 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:38:58.486397 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:58.485974 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" podUID="d81b97a5-fd37-4296-8c16-94f46421d4a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 01 22:38:59.485189 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:59.485155 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:38:59.485383 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:38:59.485216 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" podUID="d81b97a5-fd37-4296-8c16-94f46421d4a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 01 22:39:00.485779 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:00.485740 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 01 22:39:00.486216 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:00.485805 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" podUID="d81b97a5-fd37-4296-8c16-94f46421d4a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 01 22:39:01.667125 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:01.667031 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:39:01.667592 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:01.667361 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:39:01.668319 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:01.668300 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:39:08.027858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.021638 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv"] Nov 01 22:39:08.116433 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.099526 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv"] Nov 01 22:39:08.116433 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.099950 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:08.135946 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.135913 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwwfc\" (UniqueName: \"kubernetes.io/projected/5064b29e-328a-4a42-b129-1a6eba585a76-kube-api-access-nwwfc\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:08.136151 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.136133 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-credential-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:08.136313 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.136297 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-workload-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:08.136507 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.136488 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5064b29e-328a-4a42-b129-1a6eba585a76-istiod-ca-cert\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:08.136625 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.136610 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-istio-data\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:08.136906 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.136887 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-istio-envoy\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:08.137117 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.137103 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-workload-certs\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:08.137228 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.137219 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5064b29e-328a-4a42-b129-1a6eba585a76-istio-podinfo\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:08.137299 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.137291 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5064b29e-328a-4a42-b129-1a6eba585a76-istio-token\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:08.159230 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.159203 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j"] Nov 01 22:39:08.169083 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.169042 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j"] Nov 01 22:39:08.169256 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.169186 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:08.238349 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.238318 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd446c47-d4df-4f36-8eac-541975b1375f-istio-podinfo\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:08.238349 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.238354 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjswv\" (UniqueName: \"kubernetes.io/projected/bd446c47-d4df-4f36-8eac-541975b1375f-kube-api-access-gjswv\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:08.238569 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.238398 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-istio-envoy\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:08.238569 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.238452 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-workload-certs\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:08.238569 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.238514 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-workload-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:08.238696 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.238586 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5064b29e-328a-4a42-b129-1a6eba585a76-istio-podinfo\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:08.238696 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.238646 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5064b29e-328a-4a42-b129-1a6eba585a76-istio-token\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:08.238696 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.238686 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-istio-data\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:08.238852 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.238723 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-credential-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:08.238919 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.238849 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwwfc\" (UniqueName: \"kubernetes.io/projected/5064b29e-328a-4a42-b129-1a6eba585a76-kube-api-access-nwwfc\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:08.238919 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.238902 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd446c47-d4df-4f36-8eac-541975b1375f-istiod-ca-cert\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:08.239070 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.238920 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-workload-certs\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:08.239070 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.238955 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-credential-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:08.239070 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.239034 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-workload-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:08.239240 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.239086 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-workload-certs\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:08.239240 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.239135 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5064b29e-328a-4a42-b129-1a6eba585a76-istiod-ca-cert\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:08.239240 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.239165 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-istio-data\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:08.239240 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.239211 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-istio-envoy\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:08.239240 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.239235 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd446c47-d4df-4f36-8eac-541975b1375f-istio-token\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:08.239500 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.239268 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-credential-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:08.239500 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.239336 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-workload-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:08.239500 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.239392 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-istio-data\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:08.239946 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.239923 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5064b29e-328a-4a42-b129-1a6eba585a76-istiod-ca-cert\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:08.241628 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.241599 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-istio-envoy\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:08.241913 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.241893 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5064b29e-328a-4a42-b129-1a6eba585a76-istio-podinfo\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:08.251768 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.251746 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5064b29e-328a-4a42-b129-1a6eba585a76-istio-token\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:08.252052 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.252032 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwwfc\" (UniqueName: \"kubernetes.io/projected/5064b29e-328a-4a42-b129-1a6eba585a76-kube-api-access-nwwfc\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:08.341424 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.341323 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd446c47-d4df-4f36-8eac-541975b1375f-istio-podinfo\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:08.341424 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.341368 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjswv\" (UniqueName: \"kubernetes.io/projected/bd446c47-d4df-4f36-8eac-541975b1375f-kube-api-access-gjswv\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:08.341424 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.341420 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-workload-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:08.341731 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.341461 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-istio-data\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:08.341731 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.341496 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-credential-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:08.341731 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.341525 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd446c47-d4df-4f36-8eac-541975b1375f-istiod-ca-cert\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:08.341731 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.341610 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-workload-certs\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:08.341731 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.341680 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-istio-envoy\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:08.341731 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.341706 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd446c47-d4df-4f36-8eac-541975b1375f-istio-token\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:08.342081 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.341932 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-workload-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:08.342081 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.342044 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-credential-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:08.342205 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.342084 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-istio-data\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:08.342205 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.342163 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-workload-certs\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:08.342510 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.342487 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd446c47-d4df-4f36-8eac-541975b1375f-istiod-ca-cert\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:08.344242 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.344208 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-istio-envoy\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:08.344354 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.344274 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd446c47-d4df-4f36-8eac-541975b1375f-istio-podinfo\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:08.349941 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.349917 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjswv\" (UniqueName: \"kubernetes.io/projected/bd446c47-d4df-4f36-8eac-541975b1375f-kube-api-access-gjswv\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:08.350096 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.350078 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd446c47-d4df-4f36-8eac-541975b1375f-istio-token\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:08.485411 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.485367 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:08.495708 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.495683 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:08.617782 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.617708 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb"] Nov 01 22:39:08.623350 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.623323 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:08.635769 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.635738 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb"] Nov 01 22:39:08.653219 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.653171 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-workload-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:08.653478 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.653352 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f200df-709a-461d-b36c-af5003f8857a-istiod-ca-cert\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:08.653866 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.653476 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-istio-envoy\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:08.653973 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.653881 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-istio-data\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:08.653973 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.653923 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f200df-709a-461d-b36c-af5003f8857a-istio-podinfo\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:08.654091 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.653981 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c49c\" (UniqueName: \"kubernetes.io/projected/10f200df-709a-461d-b36c-af5003f8857a-kube-api-access-8c49c\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:08.654091 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.654043 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-credential-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:08.654208 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.654093 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-workload-certs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:08.654264 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.654214 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f200df-709a-461d-b36c-af5003f8857a-istio-token\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:08.755747 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.755707 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f200df-709a-461d-b36c-af5003f8857a-istiod-ca-cert\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:08.755977 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.755789 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-istio-envoy\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:08.755977 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.755815 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-istio-data\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:08.755977 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.755887 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f200df-709a-461d-b36c-af5003f8857a-istio-podinfo\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:08.756169 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.756034 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c49c\" (UniqueName: \"kubernetes.io/projected/10f200df-709a-461d-b36c-af5003f8857a-kube-api-access-8c49c\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:08.756169 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.756133 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-credential-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:08.756283 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.756187 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-workload-certs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:08.756342 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.756292 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f200df-709a-461d-b36c-af5003f8857a-istio-token\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:08.756397 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.756361 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-workload-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:08.756618 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.756593 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-workload-certs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:08.756777 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.756667 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f200df-709a-461d-b36c-af5003f8857a-istiod-ca-cert\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:08.756900 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.756792 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-workload-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:08.756900 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.756862 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-credential-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:08.757114 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.757086 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-istio-data\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:08.759726 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.759700 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f200df-709a-461d-b36c-af5003f8857a-istio-podinfo\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:08.759876 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.759725 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-istio-envoy\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:08.766757 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.766730 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c49c\" (UniqueName: \"kubernetes.io/projected/10f200df-709a-461d-b36c-af5003f8857a-kube-api-access-8c49c\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:08.778446 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.778410 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f200df-709a-461d-b36c-af5003f8857a-istio-token\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:08.806074 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.806046 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv"] Nov 01 22:39:08.808442 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.808413 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j"] Nov 01 22:39:08.808905 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.808867 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:39:08.809018 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.808940 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:39:08.809018 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.808977 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:39:08.810347 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.810317 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:39:08.810435 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.810378 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:39:08.810435 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.810406 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:39:08.876036 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.876011 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx"] Nov 01 22:39:08.883131 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.883105 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:08.902655 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.900350 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx"] Nov 01 22:39:08.953869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:08.950150 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:09.012248 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.012118 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istio-token\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:09.012621 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.012593 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-workload-certs\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:09.012995 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.012973 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istiod-ca-cert\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:09.013329 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.013173 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k6k2\" (UniqueName: \"kubernetes.io/projected/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-kube-api-access-8k6k2\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:09.013523 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.013493 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istio-podinfo\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:09.021037 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.021006 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istio-data\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:09.059863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.046587 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-credential-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:09.059863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.046904 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-workload-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:09.059863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.047071 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istio-envoy\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:09.175363 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.175331 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istiod-ca-cert\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:09.175513 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.175372 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k6k2\" (UniqueName: \"kubernetes.io/projected/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-kube-api-access-8k6k2\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:09.175513 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.175414 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istio-podinfo\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:09.175647 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.175549 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istio-data\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:09.175945 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.175923 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-credential-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:09.176043 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.175961 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-workload-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:09.176043 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.176004 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istio-envoy\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:09.176145 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.176116 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istio-token\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:09.176208 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.176162 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-workload-certs\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:09.176628 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.176608 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-credential-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:09.177285 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.176795 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-workload-certs\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:09.177692 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.177672 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-workload-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:09.178858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.178818 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istio-data\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:09.180119 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.180098 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istiod-ca-cert\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:09.200146 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.200111 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k6k2\" (UniqueName: \"kubernetes.io/projected/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-kube-api-access-8k6k2\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:09.206216 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.206179 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istio-token\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:09.221652 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.221607 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istio-envoy\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:09.226257 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.225344 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istio-podinfo\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:09.400078 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.399454 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb"] Nov 01 22:39:09.410067 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.410022 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:39:09.410174 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.410127 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:39:09.410215 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.410172 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:39:09.505407 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.505380 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:09.510236 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.510204 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" event={"ID":"10f200df-709a-461d-b36c-af5003f8857a","Type":"ContainerStarted","Data":"7324ea0bb565f61d6aecae42333b8507d62b46a1af4f3c5fcdb953790415e2b7"} Nov 01 22:39:09.512488 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.512458 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" event={"ID":"5064b29e-328a-4a42-b129-1a6eba585a76","Type":"ContainerStarted","Data":"9c94aaab568401c9ad54dc83eacfe4bf61f2c6d2c5e4e4f857e21faf2131835f"} Nov 01 22:39:09.512613 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.512519 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" event={"ID":"5064b29e-328a-4a42-b129-1a6eba585a76","Type":"ContainerStarted","Data":"34ace9d99df0b22d7207106bbe7ca8a13ffa163f3f7fd8b07daff3d47234d172"} Nov 01 22:39:09.515011 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.514980 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" event={"ID":"bd446c47-d4df-4f36-8eac-541975b1375f","Type":"ContainerStarted","Data":"1db7a4eb79dc4fa252d00ca3b7c8a6e51dec5d4052867f632e8cca4a6bb3dc97"} Nov 01 22:39:09.515130 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.515030 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" event={"ID":"bd446c47-d4df-4f36-8eac-541975b1375f","Type":"ContainerStarted","Data":"62e39f53801050cbef9d78e84e492e9aaf5dbebacaa97439f2e8ffcdec052c0a"} Nov 01 22:39:09.580140 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.580060 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" podStartSLOduration=1.580039097 podStartE2EDuration="1.580039097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:09.570778356 +0000 UTC m=+917.975368990" watchObservedRunningTime="2026-11-01 22:39:09.580039097 +0000 UTC m=+917.984629730" Nov 01 22:39:09.583891 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.582873 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" podStartSLOduration=2.58285837 podStartE2EDuration="2.58285837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:09.547448407 +0000 UTC m=+917.952039029" watchObservedRunningTime="2026-11-01 22:39:09.58285837 +0000 UTC m=+917.987449022" Nov 01 22:39:09.832848 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.832805 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx"] Nov 01 22:39:09.838127 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.838095 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:39:09.838220 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.838167 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:39:09.838220 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:09.838200 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:39:10.518857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:10.490313 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:10.519544 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:10.519518 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:10.519678 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:10.490994 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:39:10.519813 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:10.519792 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" podUID="5064b29e-328a-4a42-b129-1a6eba585a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 01 22:39:10.520216 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:10.512056 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:39:10.520366 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:10.520344 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" podUID="bd446c47-d4df-4f36-8eac-541975b1375f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 22:39:10.730107 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:10.728968 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" event={"ID":"10f200df-709a-461d-b36c-af5003f8857a","Type":"ContainerStarted","Data":"e26f2bfa4253e58d6629aa228ffc20d6b5b450973210b659f223520c8c164596"} Nov 01 22:39:10.730107 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:10.729038 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:39:10.733855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:10.733632 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" event={"ID":"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef","Type":"ContainerStarted","Data":"6d48c5c86ec2056e63654073d273714adca2a4cfdddc093daf449a807b2b2017"} Nov 01 22:39:10.733855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:10.733686 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" event={"ID":"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef","Type":"ContainerStarted","Data":"38b8cd760b55f1ee2456ef389e4de7cc98d3377634ef30cb6bc3c2602583e6d0"} Nov 01 22:39:10.816465 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:10.815895 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" podStartSLOduration=2.81587341 podStartE2EDuration="2.81587341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:10.754339648 +0000 UTC m=+919.158930281" watchObservedRunningTime="2026-11-01 22:39:10.81587341 +0000 UTC m=+919.220464042" Nov 01 22:39:10.903858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:10.901187 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" podStartSLOduration=2.901164172 podStartE2EDuration="2.901164172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:10.802332524 +0000 UTC m=+919.206923170" watchObservedRunningTime="2026-11-01 22:39:10.901164172 +0000 UTC m=+919.305754805" Nov 01 22:39:10.963856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:10.962924 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:10.963856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:10.963055 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 22:39:10.963856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:10.963104 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" podUID="10f200df-709a-461d-b36c-af5003f8857a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 22:39:11.507910 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:11.507101 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:39:11.507910 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:11.507176 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" podUID="5064b29e-328a-4a42-b129-1a6eba585a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 01 22:39:11.508160 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:11.507923 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:11.508691 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:11.508661 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:39:11.508789 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:11.508724 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" podUID="bd446c47-d4df-4f36-8eac-541975b1375f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 22:39:11.509812 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:11.509788 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 22:39:11.510032 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:11.510012 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" podUID="0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 22:39:11.907906 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:11.902231 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:11.965061 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:11.964540 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 22:39:11.965061 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:11.964606 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" podUID="10f200df-709a-461d-b36c-af5003f8857a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 22:39:12.028860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:12.021014 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:12.160919 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:12.160339 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:12.514859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:12.502149 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:39:12.541870 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:12.527934 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 01 22:39:12.541870 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:12.528012 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" podUID="5064b29e-328a-4a42-b129-1a6eba585a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 01 22:39:12.541870 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:12.528314 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 22:39:12.541870 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:12.528345 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" podUID="0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 22:39:12.541870 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:12.528409 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:12.541870 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:12.531074 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:39:12.541870 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:12.531114 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" podUID="bd446c47-d4df-4f36-8eac-541975b1375f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 22:39:12.571874 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:12.549938 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:12.594164 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:12.594126 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:12.914918 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:12.912155 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 01 22:39:12.914918 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:12.912559 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:39:12.981858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:12.940491 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:39:12.981858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:12.973993 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 22:39:12.981858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:12.974058 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" podUID="10f200df-709a-461d-b36c-af5003f8857a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 22:39:13.376939 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:13.205713 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:13.376939 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:13.206300 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:39:13.376939 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:13.212013 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 01 22:39:13.376939 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:13.276706 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 22:39:13.376939 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:13.352448 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:39:13.491861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:13.488205 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:39:13.492152 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:13.492115 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" podUID="5064b29e-328a-4a42-b129-1a6eba585a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 01 22:39:13.513528 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:13.513494 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:39:13.513756 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:13.513727 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" podUID="bd446c47-d4df-4f36-8eac-541975b1375f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 22:39:13.552071 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:13.551813 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:39:13.577259 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:13.552263 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:13.577259 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:13.552874 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:39:13.577259 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:13.552941 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:39:13.577259 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:13.554603 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:13.577259 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:13.575362 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:39:13.577259 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:13.576627 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 22:39:13.577259 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:13.576677 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" podUID="0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 22:39:13.606383 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:13.606333 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:13.609505 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:13.609477 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:39:13.831286 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:13.830396 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:13.916008 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:13.915945 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:13.964895 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:13.951070 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:13.994448 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:13.994414 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 22:39:14.011999 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:14.011952 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" podUID="10f200df-709a-461d-b36c-af5003f8857a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 22:39:14.290250 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:14.290207 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:14.305890 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:14.291560 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 22:39:14.310193 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:14.306701 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 01 22:39:14.379246 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:14.379212 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:39:14.572916 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:14.566134 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:39:14.660987 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:14.660944 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:39:14.663760 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:14.663715 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:39:14.664011 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:14.663980 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" podUID="bd446c47-d4df-4f36-8eac-541975b1375f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 22:39:14.669856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:14.666349 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 22:39:14.670093 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:14.670062 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" podUID="0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 22:39:14.702903 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:14.690681 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:39:14.702903 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:14.690751 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" podUID="5064b29e-328a-4a42-b129-1a6eba585a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 01 22:39:14.736867 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:14.732988 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:39:14.758224 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:14.758181 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 01 22:39:14.759163 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:14.759110 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:39:14.933047 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:14.932980 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 01 22:39:15.004882 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:14.972170 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:15.014968 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:15.014901 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 22:39:15.018987 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:15.018924 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" podUID="10f200df-709a-461d-b36c-af5003f8857a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 22:39:15.205957 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:15.191336 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:15.493304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:15.493203 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:15.675859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:15.674952 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:15.800847 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:15.800256 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:39:15.800847 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:15.800435 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": context deadline exceeded" Nov 01 22:39:15.800847 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:15.800568 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:39:15.800847 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:15.800660 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 01 22:39:15.805855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:15.805038 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:15.805855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:15.805356 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 22:39:15.827225 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:15.827185 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:39:15.858883 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:15.858824 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" podUID="5064b29e-328a-4a42-b129-1a6eba585a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 01 22:39:15.862929 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:15.828021 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 22:39:15.863175 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:15.863146 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" podUID="0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 22:39:15.867922 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:15.828067 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:39:15.872865 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:15.868092 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" podUID="bd446c47-d4df-4f36-8eac-541975b1375f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 22:39:15.909997 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:15.909951 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:15.981193 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:15.981134 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:16.006996 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:16.006930 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:16.007706 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:16.007679 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 22:39:16.007916 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:16.007893 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" podUID="10f200df-709a-461d-b36c-af5003f8857a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 22:39:16.518863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:16.498794 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:16.518863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:16.500814 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:39:16.518863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:16.500918 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" podUID="5064b29e-328a-4a42-b129-1a6eba585a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 01 22:39:16.518863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:16.501757 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:39:16.518863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:16.501792 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" podUID="bd446c47-d4df-4f36-8eac-541975b1375f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 22:39:16.518863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:16.518726 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 22:39:16.518863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:16.518779 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" podUID="0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 22:39:16.692860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:16.675934 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:16.864071 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:16.812745 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:16.864071 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:16.813072 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:16.864071 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:16.815276 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:16.864071 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:16.815551 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:39:16.864071 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:16.815805 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:16.900859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:16.875788 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:16.975946 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:16.968349 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 22:39:16.975946 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:16.968404 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" podUID="10f200df-709a-461d-b36c-af5003f8857a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 22:39:16.987391 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:16.986942 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:17.062857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:17.022542 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:39:17.515856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:17.490309 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:39:17.515856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:17.490373 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" podUID="5064b29e-328a-4a42-b129-1a6eba585a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 01 22:39:17.515856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:17.507438 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:39:17.515856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:17.508424 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 22:39:17.515856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:17.508461 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" podUID="0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 22:39:17.515856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:17.508701 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:39:17.515856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:17.508724 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" podUID="bd446c47-d4df-4f36-8eac-541975b1375f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 22:39:17.676568 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:17.676524 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:17.736860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:17.720006 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:17.736860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:17.720078 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" podUID="d81b97a5-fd37-4296-8c16-94f46421d4a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:17.736860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:17.720782 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:17.736860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:17.720860 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" podUID="a6d034ba-eb48-4ce8-b349-e2e816a123c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:17.818041 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:17.817927 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:17.818662 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:17.818626 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 22:39:17.847862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:17.823322 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 01 22:39:17.851856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:17.828295 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 01 22:39:17.873867 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:39:17.842074 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:39:17.874127 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:17.868523 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:17.938985 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:17.938932 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:17.956300 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:17.956259 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 22:39:17.956584 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:17.956555 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" podUID="10f200df-709a-461d-b36c-af5003f8857a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 22:39:17.993085 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:17.993040 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:18.022986 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:18.022935 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:18.064082 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:18.064038 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:18.064380 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:18.064350 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" podUID="5ec80ece-abb9-45b5-a34e-45bbb2d90eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:18.173861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:18.172961 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:18.518764 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:18.517035 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:39:18.518764 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:18.517097 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" podUID="5064b29e-328a-4a42-b129-1a6eba585a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 01 22:39:18.518764 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:18.517488 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:39:18.518764 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:18.518349 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:39:18.518764 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:18.518384 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" podUID="bd446c47-d4df-4f36-8eac-541975b1375f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 22:39:18.518764 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:18.518634 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 22:39:18.518764 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:18.518663 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" podUID="0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 22:39:18.799110 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:18.798921 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:39:18.826855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:18.823073 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 22:39:18.826855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:18.823525 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:39:18.860014 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:18.859956 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:18.885321 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:18.863047 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:18.939994 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:18.939942 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:18.969066 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:18.969025 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 22:39:18.969560 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:18.969525 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" podUID="10f200df-709a-461d-b36c-af5003f8857a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 22:39:19.014718 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:19.014674 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 01 22:39:19.029648 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:19.029600 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:19.312958 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:19.312919 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:19.313242 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:19.313211 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" podUID="dccd06ae-f4d0-485f-abe7-89cc6e34a431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:19.394198 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:19.394161 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:19.394472 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:19.394443 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" podUID="5913d691-0d87-4599-b176-d132888e187e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:19.489384 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:19.489350 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:39:19.489694 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:19.489648 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" podUID="5064b29e-328a-4a42-b129-1a6eba585a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 01 22:39:19.519040 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:19.519007 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:39:19.519305 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:19.519272 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" podUID="bd446c47-d4df-4f36-8eac-541975b1375f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 22:39:19.519511 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:19.519484 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:19.520559 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:19.520534 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 22:39:19.520738 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:19.520720 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" podUID="0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 22:39:19.805856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:19.804406 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded" Nov 01 22:39:19.824547 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:19.824495 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:19.993326 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:19.826928 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:19.997161 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:19.827744 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:20.002168 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:19.885746 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:20.002637 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:19.887093 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 01 22:39:20.152306 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:19.993030 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:39:20.188391 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:20.039290 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:20.188849 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:20.039326 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 01 22:39:20.189208 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:20.071479 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 22:39:20.189368 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:20.189338 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" podUID="10f200df-709a-461d-b36c-af5003f8857a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 22:39:20.497046 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:20.497010 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:39:20.497385 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:20.497356 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" podUID="bd446c47-d4df-4f36-8eac-541975b1375f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 22:39:20.497935 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:20.497285 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:39:20.508901 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:20.498073 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" podUID="5064b29e-328a-4a42-b129-1a6eba585a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 01 22:39:20.509742 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:20.509159 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 22:39:20.510036 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:20.510010 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" podUID="0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 22:39:20.521962 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:20.521916 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:20.961581 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:20.961535 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 22:39:20.962323 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:20.962286 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" podUID="10f200df-709a-461d-b36c-af5003f8857a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 22:39:20.996899 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:20.996808 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:39:20.997415 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:20.997161 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:39:21.021918 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:21.003259 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:21.022215 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:21.022177 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:21.081343 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:21.080980 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:21.081343 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:21.081051 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" podUID="b8318b18-986e-4435-8518-42d01c66b6b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:21.153015 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:21.152974 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 01 22:39:21.195234 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:21.193949 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:21.195234 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:21.194353 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:39:21.195234 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:21.194627 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:39:21.371082 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:21.370959 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:21.382495 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:21.382434 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:21.383363 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:21.382822 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" podUID="324f88f5-9089-49ad-9a07-22abe2a0e98f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:21.486873 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:21.486825 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:39:21.487184 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:21.487139 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" podUID="5064b29e-328a-4a42-b129-1a6eba585a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 01 22:39:21.498029 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:21.497998 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:39:21.498308 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:21.498273 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" podUID="bd446c47-d4df-4f36-8eac-541975b1375f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 22:39:21.516870 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:21.513168 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 22:39:21.516870 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:21.513224 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" podUID="0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 22:39:21.528870 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:21.525919 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:21.971969 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:21.961254 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 22:39:21.971969 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:21.961314 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" podUID="10f200df-709a-461d-b36c-af5003f8857a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 22:39:22.113856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:22.112748 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 22:39:22.170975 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:22.155121 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:39:22.170975 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:22.155490 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:22.170975 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:22.157747 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:22.306942 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:22.204287 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:22.306942 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:22.204661 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 01 22:39:22.306942 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:22.204966 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 01 22:39:22.306942 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:22.205216 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:39:22.514258 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:22.514220 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 22:39:22.514529 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:22.514501 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" podUID="0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 22:39:22.535883 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:22.531359 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:39:22.535883 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:22.531410 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" podUID="5064b29e-328a-4a42-b129-1a6eba585a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 01 22:39:22.535883 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:22.531476 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:22.538438 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:22.538408 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:39:22.538681 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:22.538657 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" podUID="bd446c47-d4df-4f36-8eac-541975b1375f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 22:39:23.047572 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:23.046443 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:39:23.059057 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:23.059025 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 22:39:23.059327 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:23.059297 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" podUID="10f200df-709a-461d-b36c-af5003f8857a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 22:39:23.146326 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:23.146281 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:23.164863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:23.158794 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:23.173214 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:23.158846 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:39:23.205279 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:23.205232 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:39:23.205613 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:23.205580 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:23.205780 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:23.205755 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:23.206006 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:23.205149 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:23.658199 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:23.658148 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:23.704513 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:23.704473 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:23.719984 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:23.719952 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 22:39:23.759137 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:23.759099 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" podUID="0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 22:39:23.762169 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:23.761898 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:39:23.783180 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:23.783145 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:39:23.783451 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:23.783422 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" podUID="bd446c47-d4df-4f36-8eac-541975b1375f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 22:39:23.807128 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:23.807083 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:23.843362 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:23.843317 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:39:23.843642 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:23.843592 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" podUID="5064b29e-328a-4a42-b129-1a6eba585a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 01 22:39:23.976546 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:23.976510 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 22:39:23.977510 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:23.977476 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" podUID="10f200df-709a-461d-b36c-af5003f8857a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 22:39:24.105894 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:24.089959 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:24.162884 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:24.161919 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 22:39:24.189186 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:24.189141 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:24.190174 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:24.189732 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:24.218332 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:24.218283 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:39:24.259908 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:24.230651 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:39:24.268856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:24.230731 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 01 22:39:24.300969 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:24.232570 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 01 22:39:24.489152 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:24.423744 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:24.511096 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:24.511006 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:39:24.538153 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:24.514940 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:39:24.538464 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:24.538414 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" podUID="5064b29e-328a-4a42-b129-1a6eba585a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 01 22:39:24.538879 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:24.516181 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:24.567873 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:24.516510 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 22:39:24.592910 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:24.579826 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" podUID="0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 22:39:24.592910 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:24.538986 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" podUID="bd446c47-d4df-4f36-8eac-541975b1375f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 22:39:24.772620 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:24.772471 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:24.874905 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:24.874874 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:39:24.964740 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:24.964702 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 22:39:24.965420 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:24.965388 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" podUID="10f200df-709a-461d-b36c-af5003f8857a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 22:39:25.034198 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:25.025454 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:39:25.175013 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:25.157940 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:25.175013 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:25.158385 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:39:25.175013 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:25.165917 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded" Nov 01 22:39:25.231870 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:25.194994 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:39:25.231870 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:25.208547 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:25.265544 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:25.265258 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:39:25.306910 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:25.304367 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 01 22:39:25.377717 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:25.364966 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 01 22:39:25.537450 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:25.529734 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 22:39:25.592107 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:25.591992 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" podUID="bd446c47-d4df-4f36-8eac-541975b1375f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 22:39:25.597991 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:25.597941 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 01 22:39:25.598260 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:25.598219 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" podUID="5064b29e-328a-4a42-b129-1a6eba585a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 01 22:39:25.623921 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:25.598472 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:39:25.624120 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:25.623981 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:39:25.624236 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:25.619480 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 22:39:25.624427 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:25.624404 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" podUID="0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 22:39:26.132629 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:26.117939 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 22:39:26.205851 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:26.190633 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 22:39:26.205851 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:26.190717 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" podUID="10f200df-709a-461d-b36c-af5003f8857a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 22:39:26.224860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:26.209154 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:39:26.224860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:26.216084 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 01 22:39:26.277895 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:26.272933 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:26.324312 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:26.323618 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:26.365962 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:26.365918 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:26.409359 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:26.409269 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:39:26.460218 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:26.460191 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:39:26.515575 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:26.515528 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 22:39:26.516132 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:26.516091 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" podUID="0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 22:39:26.554612 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:26.554532 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:39:26.796236 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:26.781530 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:39:26.959953 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:26.959922 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:39:27.325259 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:27.325228 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:27.361192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:27.361160 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 22:39:27.522345 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:27.522304 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:27.525660 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:27.525616 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" podUID="5064b29e-328a-4a42-b129-1a6eba585a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:27.572395 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:27.572360 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:27.581051 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:27.577638 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:27.581347 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:27.581324 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:39:27.587372 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:27.572798 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 01 22:39:27.587372 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:27.586555 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" podUID="0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 01 22:39:27.594168 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:27.594133 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:39:28.732118 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:28.732089 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:28.733434 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:28.733413 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:39:38.269885 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.269847 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7"] Nov 01 22:39:38.276102 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.276071 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:38.308795 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.308764 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7"] Nov 01 22:39:38.320378 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.319202 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml"] Nov 01 22:39:38.327010 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.326970 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:39:38.353445 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.353408 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml"] Nov 01 22:39:38.410486 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.410450 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97"] Nov 01 22:39:38.412911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.412876 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a29727f6-5841-404b-aa67-d596290c08d1-istio-token\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:39:38.413076 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.412941 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-istio-data\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:39:38.413076 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.413026 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-workload-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:39:38.413198 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.413136 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istiod-ca-cert\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:38.413198 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.413192 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8qnb\" (UniqueName: \"kubernetes.io/projected/a29727f6-5841-404b-aa67-d596290c08d1-kube-api-access-g8qnb\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:39:38.413302 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.413223 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-workload-certs\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:39:38.413351 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.413337 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istio-podinfo\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:38.413468 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.413442 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istio-data\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:38.413546 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.413506 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-workload-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:38.413603 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.413553 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istio-token\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:38.413603 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.413594 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-credential-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:39:38.413707 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.413651 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-credential-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:38.413707 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.413699 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-workload-certs\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:38.413888 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.413725 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-istio-envoy\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:39:38.413888 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.413863 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a29727f6-5841-404b-aa67-d596290c08d1-istio-podinfo\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:39:38.413976 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.413902 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istio-envoy\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:38.413976 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.413940 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a29727f6-5841-404b-aa67-d596290c08d1-istiod-ca-cert\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:39:38.413976 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.413968 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lr2r\" (UniqueName: \"kubernetes.io/projected/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-kube-api-access-9lr2r\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:38.416105 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.416089 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:38.429846 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.429800 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97"] Nov 01 22:39:38.515895 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.515822 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-workload-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:38.516164 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.515914 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/653a5287-e32f-41f0-a724-2e32dccff6a3-istio-token\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:38.516164 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.515964 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istio-token\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:38.516164 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.516005 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-credential-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:39:38.516164 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.516086 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-credential-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:38.516164 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.516148 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-istio-data\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:38.516457 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.516174 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-workload-certs\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:38.516457 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.516197 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-istio-envoy\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:39:38.516457 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.516284 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-workload-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:38.516457 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.516314 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a29727f6-5841-404b-aa67-d596290c08d1-istio-podinfo\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:39:38.516457 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.516358 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istio-envoy\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:38.516457 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.516399 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-credential-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:39:38.516457 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.516400 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-credential-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:38.516457 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.516409 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a29727f6-5841-404b-aa67-d596290c08d1-istiod-ca-cert\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:39:38.516922 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.516465 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lr2r\" (UniqueName: \"kubernetes.io/projected/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-kube-api-access-9lr2r\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:38.516922 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.516516 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-credential-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:38.516922 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.516564 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/653a5287-e32f-41f0-a724-2e32dccff6a3-istio-podinfo\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:38.516922 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.516623 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a29727f6-5841-404b-aa67-d596290c08d1-istio-token\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:39:38.516922 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.516630 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-workload-certs\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:38.516922 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.516653 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmhvb\" (UniqueName: \"kubernetes.io/projected/653a5287-e32f-41f0-a724-2e32dccff6a3-kube-api-access-rmhvb\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:38.516922 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.516889 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-istio-data\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:39:38.516922 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.516922 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-workload-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:39:38.517508 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.516967 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-istio-envoy\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:38.517508 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.517011 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/653a5287-e32f-41f0-a724-2e32dccff6a3-istiod-ca-cert\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:38.517508 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.517081 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istiod-ca-cert\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:38.517508 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.517108 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a29727f6-5841-404b-aa67-d596290c08d1-istiod-ca-cert\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:39:38.517508 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.517169 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-istio-data\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:39:38.517508 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.517236 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8qnb\" (UniqueName: \"kubernetes.io/projected/a29727f6-5841-404b-aa67-d596290c08d1-kube-api-access-g8qnb\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:39:38.517508 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.517452 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-workload-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:39:38.517508 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.517458 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-workload-certs\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:39:38.517958 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.517537 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istio-podinfo\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:38.517958 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.517641 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istio-data\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:38.517958 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.517684 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-workload-certs\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:38.517958 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.517715 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-workload-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:38.517958 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.517763 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-workload-certs\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:39:38.518232 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.518080 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istiod-ca-cert\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:38.518232 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.518151 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istio-data\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:38.519169 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.519142 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istio-envoy\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:38.519494 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.519469 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-istio-envoy\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:39:38.520308 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.520258 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a29727f6-5841-404b-aa67-d596290c08d1-istio-podinfo\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:39:38.520397 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.520339 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istio-podinfo\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:38.524932 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.524909 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istio-token\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:38.525035 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.524976 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lr2r\" (UniqueName: \"kubernetes.io/projected/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-kube-api-access-9lr2r\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:38.525422 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.525402 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a29727f6-5841-404b-aa67-d596290c08d1-istio-token\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:39:38.528377 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.528359 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8qnb\" (UniqueName: \"kubernetes.io/projected/a29727f6-5841-404b-aa67-d596290c08d1-kube-api-access-g8qnb\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:39:38.596493 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.595909 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:38.630859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.630013 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/653a5287-e32f-41f0-a724-2e32dccff6a3-istiod-ca-cert\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:38.630859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.630219 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-workload-certs\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:38.630859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.630248 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-workload-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:38.630859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.630294 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/653a5287-e32f-41f0-a724-2e32dccff6a3-istio-token\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:38.630859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.630383 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-istio-data\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:38.630859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.630482 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-credential-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:38.630859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.630522 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/653a5287-e32f-41f0-a724-2e32dccff6a3-istio-podinfo\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:38.630859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.630570 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmhvb\" (UniqueName: \"kubernetes.io/projected/653a5287-e32f-41f0-a724-2e32dccff6a3-kube-api-access-rmhvb\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:38.630859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.630634 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-istio-envoy\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:38.631474 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.631328 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-workload-certs\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:38.637196 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.632053 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/653a5287-e32f-41f0-a724-2e32dccff6a3-istiod-ca-cert\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:38.637196 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.632328 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-workload-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:38.637196 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.632732 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-credential-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:38.654861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.647726 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-istio-data\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:38.661862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.655970 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/653a5287-e32f-41f0-a724-2e32dccff6a3-istio-podinfo\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:38.661862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.658174 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:39:38.678180 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.675524 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmhvb\" (UniqueName: \"kubernetes.io/projected/653a5287-e32f-41f0-a724-2e32dccff6a3-kube-api-access-rmhvb\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:38.678180 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.676737 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/653a5287-e32f-41f0-a724-2e32dccff6a3-istio-token\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:38.690495 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.690443 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-istio-envoy\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:38.745707 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.745171 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:38.970304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.970274 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml"] Nov 01 22:39:38.970807 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:39:38.970787 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29727f6_5841_404b_aa67_d596290c08d1.slice/crio-ab5fd33d5d8913dd500c6cc094b245703314c486e09cf442cff1bf22f3071870 WatchSource:0}: Error finding container ab5fd33d5d8913dd500c6cc094b245703314c486e09cf442cff1bf22f3071870: Status 404 returned error can't find the container with id ab5fd33d5d8913dd500c6cc094b245703314c486e09cf442cff1bf22f3071870 Nov 01 22:39:38.974471 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.974426 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:39:38.974594 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.974525 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:39:38.974594 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.974566 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:39:38.976808 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:39:38.976784 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode66d728f_81d0_4dd4_9d6d_1fe881bfbd54.slice/crio-73c13b0dbbd6b00d87081d85e095fc9351fa2bec1d551f6d50a8fb5cbcdb1efb WatchSource:0}: Error finding container 73c13b0dbbd6b00d87081d85e095fc9351fa2bec1d551f6d50a8fb5cbcdb1efb: Status 404 returned error can't find the container with id 73c13b0dbbd6b00d87081d85e095fc9351fa2bec1d551f6d50a8fb5cbcdb1efb Nov 01 22:39:38.976906 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.976820 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7"] Nov 01 22:39:38.983901 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.983864 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:39:38.983988 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.983967 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:39:38.984043 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:38.984028 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:39:39.072201 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.072171 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97"] Nov 01 22:39:39.073767 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:39:39.073739 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653a5287_e32f_41f0_a724_2e32dccff6a3.slice/crio-cf27aa8fd6b08cfe4ab71fc8ed2f4d3212e8a2aa540c71982f4f49bf23dbaf9e WatchSource:0}: Error finding container cf27aa8fd6b08cfe4ab71fc8ed2f4d3212e8a2aa540c71982f4f49bf23dbaf9e: Status 404 returned error can't find the container with id cf27aa8fd6b08cfe4ab71fc8ed2f4d3212e8a2aa540c71982f4f49bf23dbaf9e Nov 01 22:39:39.077651 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.077602 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:39:39.077756 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.077692 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:39:39.077756 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.077735 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:39:39.129416 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.129381 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn"] Nov 01 22:39:39.137822 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.137608 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:39.167249 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.159314 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn"] Nov 01 22:39:39.224777 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.224669 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" event={"ID":"653a5287-e32f-41f0-a724-2e32dccff6a3","Type":"ContainerStarted","Data":"cf27aa8fd6b08cfe4ab71fc8ed2f4d3212e8a2aa540c71982f4f49bf23dbaf9e"} Nov 01 22:39:39.235422 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.235143 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" event={"ID":"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54","Type":"ContainerStarted","Data":"82ea74e192e9f057cc53931fc4c8f2e06cb4dd85828862bb85b2802a78ac5170"} Nov 01 22:39:39.235777 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.235710 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" event={"ID":"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54","Type":"ContainerStarted","Data":"73c13b0dbbd6b00d87081d85e095fc9351fa2bec1d551f6d50a8fb5cbcdb1efb"} Nov 01 22:39:39.242704 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.239673 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" event={"ID":"a29727f6-5841-404b-aa67-d596290c08d1","Type":"ContainerStarted","Data":"061199c7094d1a5c86e7c1de635a87beb8ed2ae6a2eb6886736e8d8f11179152"} Nov 01 22:39:39.242704 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.239736 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" event={"ID":"a29727f6-5841-404b-aa67-d596290c08d1","Type":"ContainerStarted","Data":"ab5fd33d5d8913dd500c6cc094b245703314c486e09cf442cff1bf22f3071870"} Nov 01 22:39:39.259797 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.259721 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-istio-envoy\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:39.260151 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.260127 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/068185b8-d68f-4de6-98da-8997ac7dc6db-istio-podinfo\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:39.260616 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.260592 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6crmw\" (UniqueName: \"kubernetes.io/projected/068185b8-d68f-4de6-98da-8997ac7dc6db-kube-api-access-6crmw\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:39.260864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.260820 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/068185b8-d68f-4de6-98da-8997ac7dc6db-istiod-ca-cert\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:39.261058 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.261040 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-istio-data\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:39.261268 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.261205 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/068185b8-d68f-4de6-98da-8997ac7dc6db-istio-token\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:39.261616 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.261595 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-credential-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:39.261886 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.261868 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-workload-certs\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:39.263475 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.263452 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-workload-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:39.303085 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.302968 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" podStartSLOduration=1.3029472850000001 podStartE2EDuration="1.302947285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:39.273151064 +0000 UTC m=+947.677741708" watchObservedRunningTime="2026-11-01 22:39:39.302947285 +0000 UTC m=+947.707537920" Nov 01 22:39:39.365008 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.364967 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-credential-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:39.365344 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.365325 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-workload-certs\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:39.365514 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.365485 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-credential-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:39.366064 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.365553 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-workload-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:39.366241 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.365786 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-workload-certs\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:39.366428 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.365851 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-workload-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:39.366637 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.366618 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-istio-envoy\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:39.367741 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.367723 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/068185b8-d68f-4de6-98da-8997ac7dc6db-istio-podinfo\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:39.368286 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.368268 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6crmw\" (UniqueName: \"kubernetes.io/projected/068185b8-d68f-4de6-98da-8997ac7dc6db-kube-api-access-6crmw\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:39.368434 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.368421 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/068185b8-d68f-4de6-98da-8997ac7dc6db-istiod-ca-cert\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:39.369180 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.369165 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-istio-data\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:39.369582 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.369552 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/068185b8-d68f-4de6-98da-8997ac7dc6db-istio-token\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:39.369667 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.369096 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/068185b8-d68f-4de6-98da-8997ac7dc6db-istiod-ca-cert\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:39.369667 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.369493 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-istio-data\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:39.370199 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.370178 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-istio-envoy\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:39.372726 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.372704 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/068185b8-d68f-4de6-98da-8997ac7dc6db-istio-podinfo\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:39.377648 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.377614 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/068185b8-d68f-4de6-98da-8997ac7dc6db-istio-token\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:39.377943 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.377918 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6crmw\" (UniqueName: \"kubernetes.io/projected/068185b8-d68f-4de6-98da-8997ac7dc6db-kube-api-access-6crmw\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:39.566941 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.563193 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:39.601532 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.598151 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:39.661602 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.661524 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:39:39.930801 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.930751 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" podStartSLOduration=1.930735437 podStartE2EDuration="1.930735437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:39.301883944 +0000 UTC m=+947.706474572" watchObservedRunningTime="2026-11-01 22:39:39.930735437 +0000 UTC m=+948.335326068" Nov 01 22:39:39.955368 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:39.955322 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn"] Nov 01 22:39:39.974658 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:39:39.974625 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod068185b8_d68f_4de6_98da_8997ac7dc6db.slice/crio-581789e06f96c3b7a4db6513c92ea3fed56d68de13878d68c5d0864ec1f4d2d3 WatchSource:0}: Error finding container 581789e06f96c3b7a4db6513c92ea3fed56d68de13878d68c5d0864ec1f4d2d3: Status 404 returned error can't find the container with id 581789e06f96c3b7a4db6513c92ea3fed56d68de13878d68c5d0864ec1f4d2d3 Nov 01 22:39:40.006116 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:40.006059 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:39:40.006420 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:40.006398 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:39:40.006639 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:40.006619 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:39:40.402325 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:40.402272 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" event={"ID":"653a5287-e32f-41f0-a724-2e32dccff6a3","Type":"ContainerStarted","Data":"157e502c7cce3de39f0fab055448d23214780b6fcee931ab5ff9f12350779cf6"} Nov 01 22:39:40.431936 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:40.431895 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" event={"ID":"068185b8-d68f-4de6-98da-8997ac7dc6db","Type":"ContainerStarted","Data":"96ca5830c624be2beed3560a891485f709c398aa7011c93c29be861aea7a7215"} Nov 01 22:39:40.434266 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:40.432467 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" event={"ID":"068185b8-d68f-4de6-98da-8997ac7dc6db","Type":"ContainerStarted","Data":"581789e06f96c3b7a4db6513c92ea3fed56d68de13878d68c5d0864ec1f4d2d3"} Nov 01 22:39:40.463685 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:40.463537 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" podStartSLOduration=2.463512229 podStartE2EDuration="2.463512229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:40.443298827 +0000 UTC m=+948.847889460" watchObservedRunningTime="2026-11-01 22:39:40.463512229 +0000 UTC m=+948.868102862" Nov 01 22:39:40.501507 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:40.501434 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" podStartSLOduration=1.5014121679999999 podStartE2EDuration="1.501412168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:40.479480263 +0000 UTC m=+948.884070898" watchObservedRunningTime="2026-11-01 22:39:40.501412168 +0000 UTC m=+948.906002800" Nov 01 22:39:40.577859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:40.569947 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:40.603147 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:40.603108 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 01 22:39:40.603400 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:40.603375 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" podUID="e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 01 22:39:40.672612 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:40.672476 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 22:39:40.672612 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:40.672547 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" podUID="a29727f6-5841-404b-aa67-d596290c08d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 22:39:40.762911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:40.752649 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:40.762911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:40.752916 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 22:39:40.762911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:40.752974 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" podUID="653a5287-e32f-41f0-a724-2e32dccff6a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 22:39:41.581858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:41.572649 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 22:39:41.581858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:41.572709 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" podUID="068185b8-d68f-4de6-98da-8997ac7dc6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 22:39:41.608858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:41.602365 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 01 22:39:41.608858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:41.602431 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" podUID="e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 01 22:39:41.662927 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:41.662891 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 22:39:41.663198 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:41.663174 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" podUID="a29727f6-5841-404b-aa67-d596290c08d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 22:39:41.819580 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:41.819543 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 22:39:41.835869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:41.835745 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" podUID="653a5287-e32f-41f0-a724-2e32dccff6a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 22:39:42.582924 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:42.573298 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 22:39:42.583615 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:42.583574 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" podUID="068185b8-d68f-4de6-98da-8997ac7dc6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 22:39:42.620460 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:42.620423 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 01 22:39:42.620750 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:42.620721 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" podUID="e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 01 22:39:42.681938 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:42.681895 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:42.693351 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:42.693316 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 22:39:42.711053 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:42.710944 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" podUID="a29727f6-5841-404b-aa67-d596290c08d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 22:39:42.741252 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:42.740446 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:42.753910 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:42.753483 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 22:39:42.753910 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:42.753556 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" podUID="653a5287-e32f-41f0-a724-2e32dccff6a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 22:39:42.781858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:42.778906 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:42.838364 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:42.838266 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:42.918989 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:42.918946 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:43.030061 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:43.023001 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:39:43.235858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:43.235684 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:43.299858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:43.295943 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:43.324859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:43.321914 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 22:39:43.600909 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:43.581364 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 22:39:43.638858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:43.607689 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" podUID="068185b8-d68f-4de6-98da-8997ac7dc6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 22:39:43.638858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:43.604083 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 01 22:39:43.638858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:43.608190 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" podUID="e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 01 22:39:43.703861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:43.665845 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 22:39:43.703861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:43.665907 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" podUID="a29727f6-5841-404b-aa67-d596290c08d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 22:39:43.770856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:43.725107 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:43.770856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:43.748207 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:39:43.820871 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:43.780575 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:43.820871 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:43.784108 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:43.820871 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:43.812177 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:39:43.926598 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:43.897248 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:43.926598 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:43.897949 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 22:39:43.926598 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:43.898331 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:39:43.956041 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:43.955987 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" podUID="653a5287-e32f-41f0-a724-2e32dccff6a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 22:39:43.956403 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:43.956364 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:44.456709 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:44.362207 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:44.557902 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:44.554646 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:39:44.724907 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:44.711664 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 01 22:39:44.724907 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:44.711725 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" podUID="e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 01 22:39:44.891031 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:44.890991 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 22:39:44.966327 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:44.966269 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:44.966901 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:44.966869 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:45.110132 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:45.110008 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:45.137424 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:45.137262 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:45.138534 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:45.138503 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:45.396492 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:45.270225 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 22:39:45.396780 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:45.396726 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" podUID="a29727f6-5841-404b-aa67-d596290c08d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 22:39:45.396939 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:45.270885 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 22:39:45.397056 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:45.397037 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" podUID="068185b8-d68f-4de6-98da-8997ac7dc6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 22:39:45.397189 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:44.891285 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" podUID="653a5287-e32f-41f0-a724-2e32dccff6a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 22:39:45.426406 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:45.426366 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:39:45.592911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:45.568078 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:39:45.593816 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:45.593785 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 22:39:45.594044 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:45.594021 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" podUID="068185b8-d68f-4de6-98da-8997ac7dc6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 22:39:45.618857 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:39:45.610359 5011 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.712s" Nov 01 22:39:45.723966 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:45.723937 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:39:45.724235 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:45.724223 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:39:45.724813 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:45.724796 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:39:45.725022 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:45.725011 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:39:45.725911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:45.642602 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded" Nov 01 22:39:45.816859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:45.686407 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-67p7h" podUID="cf198325-a260-434e-a504-f73add235e9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Nov 01 22:39:45.943891 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:45.726208 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 22:39:45.971501 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:45.971462 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" podUID="a29727f6-5841-404b-aa67-d596290c08d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 22:39:46.000559 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:45.738158 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 01 22:39:46.000819 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:46.000790 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" podUID="e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 01 22:39:46.001038 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:45.846697 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 22:39:46.001166 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:46.001144 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" podUID="653a5287-e32f-41f0-a724-2e32dccff6a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 22:39:46.001372 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:45.686349 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:46.107860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:45.970930 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:46.135862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:46.125928 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": context deadline exceeded" Nov 01 22:39:46.151030 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:46.136190 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:39:46.207262 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:46.197467 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 01 22:39:46.207262 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:46.200206 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:46.207262 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:46.200762 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:46.384109 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:46.349442 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:46.396210 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:46.396161 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:39:46.436056 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:46.435996 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:46.624871 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:46.605775 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:46.860910 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:46.625242 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 01 22:39:46.860910 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:46.859433 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" podUID="e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 01 22:39:46.860910 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:46.735096 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 22:39:46.860910 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:46.859497 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" podUID="a29727f6-5841-404b-aa67-d596290c08d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 22:39:46.860910 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:46.782992 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 22:39:46.860910 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:46.859533 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" podUID="068185b8-d68f-4de6-98da-8997ac7dc6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 22:39:46.860910 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:46.804982 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:46.863411 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:46.863130 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 22:39:46.863726 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:46.863699 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" podUID="653a5287-e32f-41f0-a724-2e32dccff6a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 22:39:47.053976 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:47.023931 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:47.053976 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:47.024433 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:39:47.085532 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:47.084932 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:47.193974 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:47.193929 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:39:47.217246 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:47.201599 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:47.409861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:47.220467 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:39:47.409861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:47.202114 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:47.409861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:47.366930 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:47.409861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:47.398138 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:39:47.656912 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:47.631248 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:39:47.742034 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:47.686070 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 22:39:47.742317 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:47.742284 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" podUID="068185b8-d68f-4de6-98da-8997ac7dc6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 22:39:47.742450 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:47.695781 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 22:39:47.742561 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:47.742544 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" podUID="a29727f6-5841-404b-aa67-d596290c08d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 22:39:47.745980 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:47.745948 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 01 22:39:47.746179 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:47.746156 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" podUID="e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 01 22:39:47.746561 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:39:47.746534 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:39:47.775247 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:47.747125 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 22:39:47.775515 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:47.775489 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" podUID="653a5287-e32f-41f0-a724-2e32dccff6a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 22:39:47.819071 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:47.818935 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:47.819071 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:47.819007 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" podUID="d81b97a5-fd37-4296-8c16-94f46421d4a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:47.893862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:47.874669 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:47.912079 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:47.879104 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:48.030870 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:48.029181 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 01 22:39:48.086334 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:48.085348 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:48.167057 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:48.166946 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded" Nov 01 22:39:48.241583 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:48.219048 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:48.241583 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:48.222054 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:48.241583 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:48.223081 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 01 22:39:48.369269 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:48.369213 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:48.405198 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:48.405150 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:39:48.583627 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:48.583575 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 22:39:48.583980 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:48.583930 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" podUID="068185b8-d68f-4de6-98da-8997ac7dc6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 22:39:48.611799 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:48.611764 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 01 22:39:48.626587 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:48.625905 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" podUID="e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 01 22:39:48.664298 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:48.664258 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 22:39:48.664569 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:48.664538 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" podUID="a29727f6-5841-404b-aa67-d596290c08d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 22:39:48.749296 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:48.749249 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:48.751531 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:48.751501 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 22:39:48.751764 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:48.751742 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" podUID="653a5287-e32f-41f0-a724-2e32dccff6a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 22:39:48.915128 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:48.914990 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:49.033055 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:49.032985 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:49.093078 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:49.093015 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 01 22:39:49.221723 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:49.221676 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:49.225865 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:49.223920 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:49.225865 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:49.224331 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:39:49.367319 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:49.367279 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:49.367601 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:49.367572 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" podUID="5913d691-0d87-4599-b176-d132888e187e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:49.371012 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:49.370971 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:49.418322 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:49.418276 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:49.572918 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:49.572048 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 22:39:49.572918 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:49.572113 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" podUID="068185b8-d68f-4de6-98da-8997ac7dc6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 22:39:49.598859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:49.598614 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 01 22:39:49.598859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:49.598693 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" podUID="e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 01 22:39:49.641569 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:49.641524 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:49.667300 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:49.667262 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 22:39:49.667588 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:49.667556 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" podUID="a29727f6-5841-404b-aa67-d596290c08d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 22:39:49.771780 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:49.771742 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 22:39:49.772106 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:49.772073 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" podUID="653a5287-e32f-41f0-a724-2e32dccff6a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 22:39:49.773570 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:49.773540 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:39:49.917213 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:49.917109 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:50.043859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:50.034922 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 01 22:39:50.094859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:50.093933 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:50.222755 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:50.222246 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:50.226813 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:50.226776 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:50.230031 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:50.227051 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 01 22:39:50.372318 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:50.372262 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:50.431997 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:50.431949 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:50.568353 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:50.568253 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 22:39:50.568614 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:50.568584 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" podUID="068185b8-d68f-4de6-98da-8997ac7dc6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 22:39:50.599647 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:50.599592 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 01 22:39:50.600003 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:50.599945 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" podUID="e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 01 22:39:50.662339 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:50.662305 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 22:39:50.662653 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:50.662607 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" podUID="a29727f6-5841-404b-aa67-d596290c08d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 22:39:50.751860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:50.751721 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 22:39:50.751860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:50.751798 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" podUID="653a5287-e32f-41f0-a724-2e32dccff6a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 22:39:50.775855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:50.775755 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:50.969472 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:50.969421 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:39:51.036858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:51.035935 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:51.121977 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:51.121925 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:51.253060 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:51.252947 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:51.262236 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:51.260263 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:51.262236 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:51.261607 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 22:39:51.335432 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:51.334958 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:51.335432 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:51.335025 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" podUID="324f88f5-9089-49ad-9a07-22abe2a0e98f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:51.397857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:51.391935 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:51.397857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:51.395145 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:39:51.454735 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:51.454690 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:39:51.572916 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:51.572331 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 22:39:51.572916 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:51.572406 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" podUID="068185b8-d68f-4de6-98da-8997ac7dc6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 22:39:51.618856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:51.618745 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 01 22:39:51.619065 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:51.618812 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" podUID="e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 01 22:39:51.663958 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:51.663920 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 22:39:51.664195 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:51.664171 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" podUID="a29727f6-5841-404b-aa67-d596290c08d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 22:39:51.755252 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:51.755213 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 22:39:51.775954 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:51.755505 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" podUID="653a5287-e32f-41f0-a724-2e32dccff6a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 22:39:51.776413 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:51.776371 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:51.970252 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:51.970205 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:52.086859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:52.060525 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:52.125152 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:52.125089 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:52.291967 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:52.284176 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:52.295666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:52.292519 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:52.295666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:52.292919 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:39:52.399983 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:52.399934 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:39:52.467873 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:52.455395 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:52.595500 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:52.595370 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 22:39:52.595777 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:52.595731 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" podUID="068185b8-d68f-4de6-98da-8997ac7dc6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 22:39:52.598001 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:52.597977 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 01 22:39:52.598182 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:52.598161 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" podUID="e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 01 22:39:52.721680 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:52.721379 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 22:39:52.722076 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:52.722035 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" podUID="a29727f6-5841-404b-aa67-d596290c08d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 22:39:52.798664 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:52.779294 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:52.831863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:52.831233 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 22:39:52.831863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:52.831296 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" podUID="653a5287-e32f-41f0-a724-2e32dccff6a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 22:39:52.973988 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:52.973929 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:53.069860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:53.069796 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:53.130441 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:53.130400 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 01 22:39:53.310914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:53.285159 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:53.311482 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:53.304648 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:53.312068 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:53.304687 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 01 22:39:53.438006 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:53.424329 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:39:53.492170 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:53.492127 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:39:53.623944 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:53.615801 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:53.662987 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:53.662948 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:39:53.738878 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:53.732328 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 22:39:53.738878 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:53.732389 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" podUID="068185b8-d68f-4de6-98da-8997ac7dc6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 22:39:53.779891 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:53.771957 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:53.780776 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:53.780745 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 01 22:39:53.781049 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:53.781007 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" podUID="e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 01 22:39:53.782791 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:53.782755 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:53.814774 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:53.806559 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 22:39:53.814774 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:53.806628 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" podUID="a29727f6-5841-404b-aa67-d596290c08d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 22:39:53.814774 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:53.807106 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 22:39:53.814774 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:53.807144 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" podUID="653a5287-e32f-41f0-a724-2e32dccff6a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 22:39:53.814774 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:53.807195 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded" Nov 01 22:39:53.995696 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:53.995636 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:54.140681 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:54.140644 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 01 22:39:54.175455 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:54.175399 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:54.337823 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:54.337727 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:54.342678 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:54.342635 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 22:39:54.346078 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:54.346026 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:39:54.455883 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:54.450610 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:54.455883 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:54.453172 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:39:54.494937 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:54.494275 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:39:54.521100 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:54.511156 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:39:54.572859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:54.571050 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:54.575503 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:54.575473 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 22:39:54.575719 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:54.575693 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" podUID="068185b8-d68f-4de6-98da-8997ac7dc6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 22:39:54.651067 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:54.650979 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 01 22:39:54.651308 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:54.651280 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" podUID="e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 01 22:39:54.660867 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:54.660803 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 22:39:54.661139 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:54.661092 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" podUID="a29727f6-5841-404b-aa67-d596290c08d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 22:39:54.688162 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:54.685560 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:39:54.695940 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:54.695899 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:39:54.778465 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:54.778413 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 22:39:54.778678 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:54.778483 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" podUID="653a5287-e32f-41f0-a724-2e32dccff6a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 22:39:54.787979 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:54.787949 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:39:54.794859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:54.788950 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:39:54.836270 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:54.836244 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:39:55.027368 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:55.025561 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:39:55.339710 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:55.338929 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:55.350859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:55.346907 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:55.623192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:55.623084 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 22:39:55.624024 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:55.623410 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" podUID="068185b8-d68f-4de6-98da-8997ac7dc6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 22:39:55.785001 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:55.784965 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 22:39:55.787269 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:55.785219 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" podUID="653a5287-e32f-41f0-a724-2e32dccff6a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 22:39:55.841956 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:55.841914 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:55.885706 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:55.885624 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:39:55.891859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:55.886213 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:39:56.569186 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:56.569075 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 22:39:56.569186 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:56.569134 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" podUID="068185b8-d68f-4de6-98da-8997ac7dc6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 22:39:56.601800 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:56.601759 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:56.602138 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:56.602099 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" podUID="e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:56.723123 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:56.723079 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:56.723420 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:56.723386 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" podUID="a29727f6-5841-404b-aa67-d596290c08d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:57.003116 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:57.003082 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:57.004649 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:57.004625 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:39:57.734324 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:57.734277 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:57.771095 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:57.771054 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" podUID="e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:57.790615 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:57.789954 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:57.790615 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:57.790017 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" podUID="a29727f6-5841-404b-aa67-d596290c08d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:58.397266 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:58.397232 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:58.412758 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:58.412730 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:58.426095 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:58.425880 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:39:58.427169 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:58.427149 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:39:58.614489 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:58.614440 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:39:58.937328 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:39:58.937298 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 22:40:12.493751 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:12.493716 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:40:14.749972 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:14.749923 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:14.931003 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:14.930938 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:15.049976 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:15.047965 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:15.130005 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:15.129959 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:15.131439 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:15.131407 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 01 22:40:15.144979 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:15.144916 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:15.168994 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:15.168946 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:15.229945 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:15.229901 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:15.360915 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:15.359964 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:15.549859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:15.498918 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:15.749857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:15.748923 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:15.753942 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:15.753899 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:15.841085 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:15.840989 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:15.873868 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:15.844513 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:40:15.932172 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:15.932130 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:15.945422 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:15.945394 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:40:16.054917 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:16.051058 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:16.146118 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:16.136693 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:16.197856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:16.136742 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 01 22:40:16.197856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:16.195797 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 01 22:40:16.219591 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:16.180947 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:40:16.258281 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:16.258241 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:40:16.259310 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:16.259279 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:40:16.259549 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:16.259518 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:40:16.265895 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:16.261670 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:16.294857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:16.274984 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:40:16.387911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:16.376741 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:16.387911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:16.378252 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:40:16.510857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:16.503251 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:16.510857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:16.503804 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:40:16.768858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:16.768231 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:16.885897 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:16.832754 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded" Nov 01 22:40:16.982876 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:16.980471 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:17.147909 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:17.147760 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:17.172858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:17.170191 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:17.212856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:17.212050 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:17.212856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:17.212374 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:17.407768 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:17.381365 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:17.444470 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:17.414225 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:17.514868 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:17.509931 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:17.591021 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:40:17.590977 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:40:17.824990 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:17.824947 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:40:17.825579 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:17.825544 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" podUID="d81b97a5-fd37-4296-8c16-94f46421d4a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:17.854130 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:17.834656 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:40:17.894857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:17.869245 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:17.977983 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:17.977941 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:40:17.981859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:17.978230 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" podUID="a6d034ba-eb48-4ce8-b349-e2e816a123c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:18.009992 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:18.001141 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:40:18.171353 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:18.159163 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:18.171353 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:18.160389 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:40:18.183076 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:18.183027 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:18.233012 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:18.232965 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:18.234291 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:18.234259 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 01 22:40:18.452695 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:18.451376 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:40:18.535861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:18.473740 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:18.536283 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:18.495208 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:40:18.561956 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:18.561909 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" podUID="5ec80ece-abb9-45b5-a34e-45bbb2d90eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:18.591527 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:18.591479 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:40:18.857915 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:18.857165 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:19.136921 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:19.134246 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:40:19.204212 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:19.204163 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:19.206055 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:19.206021 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:40:19.257564 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:19.257516 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:19.323648 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:19.323599 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:19.516900 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:19.511567 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:40:19.622870 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:19.615538 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 22:40:19.622870 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:19.616377 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:19.686185 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:19.686138 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded" Nov 01 22:40:19.878216 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:19.878013 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:40:20.013861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:20.013096 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:40:20.013861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:20.013172 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" podUID="5913d691-0d87-4599-b176-d132888e187e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:20.174050 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:20.171434 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:20.207008 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:20.206954 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:20.316594 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:20.316549 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:20.355895 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:20.355854 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 01 22:40:20.596909 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:20.596767 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:20.631012 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:20.630959 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:20.631355 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:20.631327 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 22:40:20.668732 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:20.668528 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:20.881035 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:20.880933 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:21.075123 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:21.074510 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:40:21.158027 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:21.155617 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:40:21.166971 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:21.166939 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:40:21.171341 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:21.171311 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:40:21.174499 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:21.174467 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:40:21.174691 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:21.174680 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:40:21.216940 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:21.214915 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:40:21.279531 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:21.279505 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:40:21.303265 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:21.303237 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:40:24.144358 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:24.144324 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c"] Nov 01 22:40:24.150924 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:24.150899 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:24.172384 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:24.166591 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-88896ec5-openshift-default-dockercfg-8rzbz\"" Nov 01 22:40:24.172384 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:24.172131 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c"] Nov 01 22:40:24.293120 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:24.259951 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b6cb03-c7e7-478e-ba20-02a34c02d602-istio-podinfo\") pod \"e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:24.384190 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:24.324412 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b6cb03-c7e7-478e-ba20-02a34c02d602-istio-token\") pod \"e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:24.384190 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:24.324480 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-istio-data\") pod \"e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:24.384190 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:24.324518 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqwc4\" (UniqueName: \"kubernetes.io/projected/20b6cb03-c7e7-478e-ba20-02a34c02d602-kube-api-access-pqwc4\") pod \"e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:24.384190 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:24.326633 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-workload-socket\") pod \"e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:24.384190 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:24.326859 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-istio-envoy\") pod \"e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:24.384190 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:24.326963 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b6cb03-c7e7-478e-ba20-02a34c02d602-istiod-ca-cert\") pod \"e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:24.384190 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:24.327036 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-workload-certs\") pod \"e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:24.384190 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:24.327088 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-credential-socket\") pod \"e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:24.434855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:24.430256 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-workload-certs\") pod \"e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:24.434855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:24.430328 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-credential-socket\") pod \"e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:24.434855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:24.430474 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b6cb03-c7e7-478e-ba20-02a34c02d602-istio-podinfo\") pod \"e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:24.434855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:24.430503 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b6cb03-c7e7-478e-ba20-02a34c02d602-istio-token\") pod \"e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:24.434855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:24.430541 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-istio-data\") pod \"e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:24.434855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:24.430573 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqwc4\" (UniqueName: \"kubernetes.io/projected/20b6cb03-c7e7-478e-ba20-02a34c02d602-kube-api-access-pqwc4\") pod \"e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:24.434855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:24.430639 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-workload-socket\") pod \"e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:24.434855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:24.430813 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-istio-envoy\") pod \"e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:24.434855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:24.430917 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b6cb03-c7e7-478e-ba20-02a34c02d602-istiod-ca-cert\") pod \"e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:24.434855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:24.433029 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b6cb03-c7e7-478e-ba20-02a34c02d602-istiod-ca-cert\") pod \"e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:24.434855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:24.433319 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-workload-certs\") pod \"e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:24.434855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:24.433651 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-credential-socket\") pod \"e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:24.460855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:24.439641 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-istio-data\") pod \"e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:24.460855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:24.439884 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-workload-socket\") pod \"e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:24.488600 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:24.488569 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-istio-envoy\") pod \"e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:24.508855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:24.491805 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b6cb03-c7e7-478e-ba20-02a34c02d602-istio-podinfo\") pod \"e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:24.509688 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:24.509656 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b6cb03-c7e7-478e-ba20-02a34c02d602-istio-token\") pod \"e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:24.511663 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:24.511616 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqwc4\" (UniqueName: \"kubernetes.io/projected/20b6cb03-c7e7-478e-ba20-02a34c02d602-kube-api-access-pqwc4\") pod \"e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:24.775592 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:24.772925 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:25.180142 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:25.180099 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c"] Nov 01 22:40:25.181866 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:40:25.181805 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20b6cb03_c7e7_478e_ba20_02a34c02d602.slice/crio-df372c3e29071299577151e93f5d381eb3068f4c033c674648af8c47442783ce WatchSource:0}: Error finding container df372c3e29071299577151e93f5d381eb3068f4c033c674648af8c47442783ce: Status 404 returned error can't find the container with id df372c3e29071299577151e93f5d381eb3068f4c033c674648af8c47442783ce Nov 01 22:40:25.187868 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:25.185026 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:40:25.187868 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:25.185202 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:40:25.187868 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:25.185296 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:40:26.137859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:26.137251 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" event={"ID":"20b6cb03-c7e7-478e-ba20-02a34c02d602","Type":"ContainerStarted","Data":"81f9c40f2e3851ba5d6dfad1138b53d08cb95fcae97e665ee1d2fc6901b58797"} Nov 01 22:40:26.137859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:26.137318 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" event={"ID":"20b6cb03-c7e7-478e-ba20-02a34c02d602","Type":"ContainerStarted","Data":"df372c3e29071299577151e93f5d381eb3068f4c033c674648af8c47442783ce"} Nov 01 22:40:26.159592 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:26.159531 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" podStartSLOduration=2.159512162 podStartE2EDuration="2.159512162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:26.157073254 +0000 UTC m=+994.561663885" watchObservedRunningTime="2026-11-01 22:40:26.159512162 +0000 UTC m=+994.564102795" Nov 01 22:40:26.794057 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:26.794018 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:27.037169 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:27.037141 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:27.175855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:27.171078 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:27.179213 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:27.179151 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:40:29.171740 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:29.171705 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-88896ec5-677cd44fc5-xf2wh"] Nov 01 22:40:29.176736 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:29.176715 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-88896ec5-677cd44fc5-xf2wh" Nov 01 22:40:29.199644 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:29.199614 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-88896ec5-677cd44fc5-xf2wh"] Nov 01 22:40:29.234450 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:29.234392 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-88896ec5\"" Nov 01 22:40:29.379860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:29.322017 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0215a01-c33e-4921-b292-88ad9d3dbb31-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-88896ec5-677cd44fc5-xf2wh\" (UID: \"f0215a01-c33e-4921-b292-88ad9d3dbb31\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-88896ec5-677cd44fc5-xf2wh" Nov 01 22:40:29.379860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:29.322267 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7ncl\" (UniqueName: \"kubernetes.io/projected/f0215a01-c33e-4921-b292-88ad9d3dbb31-kube-api-access-j7ncl\") pod \"payload-pre-processing-e2e-ait-88896ec5-677cd44fc5-xf2wh\" (UID: \"f0215a01-c33e-4921-b292-88ad9d3dbb31\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-88896ec5-677cd44fc5-xf2wh" Nov 01 22:40:29.379860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:29.331938 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-88896ec5-dockercfg-b4wdm\"" Nov 01 22:40:29.500916 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:29.455244 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7ncl\" (UniqueName: \"kubernetes.io/projected/f0215a01-c33e-4921-b292-88ad9d3dbb31-kube-api-access-j7ncl\") pod \"payload-pre-processing-e2e-ait-88896ec5-677cd44fc5-xf2wh\" (UID: \"f0215a01-c33e-4921-b292-88ad9d3dbb31\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-88896ec5-677cd44fc5-xf2wh" Nov 01 22:40:29.500916 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:29.455524 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0215a01-c33e-4921-b292-88ad9d3dbb31-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-88896ec5-677cd44fc5-xf2wh\" (UID: \"f0215a01-c33e-4921-b292-88ad9d3dbb31\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-88896ec5-677cd44fc5-xf2wh" Nov 01 22:40:29.500916 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:29.456688 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0215a01-c33e-4921-b292-88ad9d3dbb31-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-88896ec5-677cd44fc5-xf2wh\" (UID: \"f0215a01-c33e-4921-b292-88ad9d3dbb31\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-88896ec5-677cd44fc5-xf2wh" Nov 01 22:40:29.567866 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:29.567365 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7ncl\" (UniqueName: \"kubernetes.io/projected/f0215a01-c33e-4921-b292-88ad9d3dbb31-kube-api-access-j7ncl\") pod \"payload-pre-processing-e2e-ait-88896ec5-677cd44fc5-xf2wh\" (UID: \"f0215a01-c33e-4921-b292-88ad9d3dbb31\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-88896ec5-677cd44fc5-xf2wh" Nov 01 22:40:29.803801 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:29.803715 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-88896ec5-677cd44fc5-xf2wh" Nov 01 22:40:30.100918 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:30.100884 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-88896ec5-677cd44fc5-xf2wh"] Nov 01 22:40:30.104228 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:40:30.104198 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0215a01_c33e_4921_b292_88ad9d3dbb31.slice/crio-ab8f85913fe5da086c864d6f8e47f1565720a2791873e6efbe0b2d5c552d3dd7 WatchSource:0}: Error finding container ab8f85913fe5da086c864d6f8e47f1565720a2791873e6efbe0b2d5c552d3dd7: Status 404 returned error can't find the container with id ab8f85913fe5da086c864d6f8e47f1565720a2791873e6efbe0b2d5c552d3dd7 Nov 01 22:40:30.553874 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:30.553133 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-88896ec5-677cd44fc5-xf2wh" event={"ID":"f0215a01-c33e-4921-b292-88ad9d3dbb31","Type":"ContainerStarted","Data":"ab8f85913fe5da086c864d6f8e47f1565720a2791873e6efbe0b2d5c552d3dd7"} Nov 01 22:40:31.279751 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:31.279710 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b0f6226e-58c9f9b8f9-nvhd6"] Nov 01 22:40:31.284639 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:31.284613 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0f6226e-58c9f9b8f9-nvhd6" Nov 01 22:40:31.292252 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:31.292216 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b0f6226e\"" Nov 01 22:40:31.296641 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:31.296612 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b0f6226e-58c9f9b8f9-nvhd6"] Nov 01 22:40:31.306816 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:31.306783 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b0f6226e-69867d597-9crx9"] Nov 01 22:40:31.311227 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:31.311201 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b0f6226e-69867d597-9crx9" Nov 01 22:40:31.324455 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:31.324409 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b0f6226e-dockercfg-svn5v\"" Nov 01 22:40:31.329711 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:31.329682 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b0f6226e-69867d597-9crx9"] Nov 01 22:40:31.402858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:31.402800 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e79b24-6ac7-474d-bb27-ec1ce218afbb-plugins-config-volume\") pod \"payload-processing-e2e-ait-b0f6226e-69867d597-9crx9\" (UID: \"84e79b24-6ac7-474d-bb27-ec1ce218afbb\") " pod="openshift-ingress/payload-processing-e2e-ait-b0f6226e-69867d597-9crx9" Nov 01 22:40:31.403435 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:31.402869 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdkvb\" (UniqueName: \"kubernetes.io/projected/baa15442-fd2c-4bdb-90c8-ff3c043ed73e-kube-api-access-pdkvb\") pod \"payload-pre-processing-e2e-ait-b0f6226e-58c9f9b8f9-nvhd6\" (UID: \"baa15442-fd2c-4bdb-90c8-ff3c043ed73e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b0f6226e-58c9f9b8f9-nvhd6" Nov 01 22:40:31.403435 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:31.402914 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65qcc\" (UniqueName: \"kubernetes.io/projected/84e79b24-6ac7-474d-bb27-ec1ce218afbb-kube-api-access-65qcc\") pod \"payload-processing-e2e-ait-b0f6226e-69867d597-9crx9\" (UID: \"84e79b24-6ac7-474d-bb27-ec1ce218afbb\") " pod="openshift-ingress/payload-processing-e2e-ait-b0f6226e-69867d597-9crx9" Nov 01 22:40:31.403435 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:31.403264 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa15442-fd2c-4bdb-90c8-ff3c043ed73e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b0f6226e-58c9f9b8f9-nvhd6\" (UID: \"baa15442-fd2c-4bdb-90c8-ff3c043ed73e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b0f6226e-58c9f9b8f9-nvhd6" Nov 01 22:40:31.505440 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:31.505408 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa15442-fd2c-4bdb-90c8-ff3c043ed73e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b0f6226e-58c9f9b8f9-nvhd6\" (UID: \"baa15442-fd2c-4bdb-90c8-ff3c043ed73e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b0f6226e-58c9f9b8f9-nvhd6" Nov 01 22:40:31.505656 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:31.505634 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e79b24-6ac7-474d-bb27-ec1ce218afbb-plugins-config-volume\") pod \"payload-processing-e2e-ait-b0f6226e-69867d597-9crx9\" (UID: \"84e79b24-6ac7-474d-bb27-ec1ce218afbb\") " pod="openshift-ingress/payload-processing-e2e-ait-b0f6226e-69867d597-9crx9" Nov 01 22:40:31.505773 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:31.505663 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdkvb\" (UniqueName: \"kubernetes.io/projected/baa15442-fd2c-4bdb-90c8-ff3c043ed73e-kube-api-access-pdkvb\") pod \"payload-pre-processing-e2e-ait-b0f6226e-58c9f9b8f9-nvhd6\" (UID: \"baa15442-fd2c-4bdb-90c8-ff3c043ed73e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b0f6226e-58c9f9b8f9-nvhd6" Nov 01 22:40:31.505773 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:31.505689 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65qcc\" (UniqueName: \"kubernetes.io/projected/84e79b24-6ac7-474d-bb27-ec1ce218afbb-kube-api-access-65qcc\") pod \"payload-processing-e2e-ait-b0f6226e-69867d597-9crx9\" (UID: \"84e79b24-6ac7-474d-bb27-ec1ce218afbb\") " pod="openshift-ingress/payload-processing-e2e-ait-b0f6226e-69867d597-9crx9" Nov 01 22:40:31.506480 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:31.506454 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa15442-fd2c-4bdb-90c8-ff3c043ed73e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b0f6226e-58c9f9b8f9-nvhd6\" (UID: \"baa15442-fd2c-4bdb-90c8-ff3c043ed73e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b0f6226e-58c9f9b8f9-nvhd6" Nov 01 22:40:31.506582 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:31.506560 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e79b24-6ac7-474d-bb27-ec1ce218afbb-plugins-config-volume\") pod \"payload-processing-e2e-ait-b0f6226e-69867d597-9crx9\" (UID: \"84e79b24-6ac7-474d-bb27-ec1ce218afbb\") " pod="openshift-ingress/payload-processing-e2e-ait-b0f6226e-69867d597-9crx9" Nov 01 22:40:31.514208 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:31.514185 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdkvb\" (UniqueName: \"kubernetes.io/projected/baa15442-fd2c-4bdb-90c8-ff3c043ed73e-kube-api-access-pdkvb\") pod \"payload-pre-processing-e2e-ait-b0f6226e-58c9f9b8f9-nvhd6\" (UID: \"baa15442-fd2c-4bdb-90c8-ff3c043ed73e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b0f6226e-58c9f9b8f9-nvhd6" Nov 01 22:40:31.514398 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:31.514206 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65qcc\" (UniqueName: \"kubernetes.io/projected/84e79b24-6ac7-474d-bb27-ec1ce218afbb-kube-api-access-65qcc\") pod \"payload-processing-e2e-ait-b0f6226e-69867d597-9crx9\" (UID: \"84e79b24-6ac7-474d-bb27-ec1ce218afbb\") " pod="openshift-ingress/payload-processing-e2e-ait-b0f6226e-69867d597-9crx9" Nov 01 22:40:31.563655 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:31.563529 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-88896ec5-677cd44fc5-xf2wh" event={"ID":"f0215a01-c33e-4921-b292-88ad9d3dbb31","Type":"ContainerStarted","Data":"294c9b5699db937f3ce89646ac91e1fab43246ffece420ef8708c76265796893"} Nov 01 22:40:31.564203 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:31.564179 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-88896ec5-677cd44fc5-xf2wh" Nov 01 22:40:31.585420 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:31.585359 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-88896ec5-677cd44fc5-xf2wh" podStartSLOduration=2.5853399599999998 podStartE2EDuration="2.58533996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:31.583192364 +0000 UTC m=+999.987782997" watchObservedRunningTime="2026-11-01 22:40:31.58533996 +0000 UTC m=+999.989930593" Nov 01 22:40:31.616183 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:31.616142 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0f6226e-58c9f9b8f9-nvhd6" Nov 01 22:40:31.659861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:31.654388 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b0f6226e-69867d597-9crx9" Nov 01 22:40:32.488151 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:32.488125 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b0f6226e-69867d597-9crx9"] Nov 01 22:40:32.488755 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:40:32.488728 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e79b24_6ac7_474d_bb27_ec1ce218afbb.slice/crio-b3e918099be1811117478d5f93cdf1331abad2767115ae0ecbe494c1a95bc438 WatchSource:0}: Error finding container b3e918099be1811117478d5f93cdf1331abad2767115ae0ecbe494c1a95bc438: Status 404 returned error can't find the container with id b3e918099be1811117478d5f93cdf1331abad2767115ae0ecbe494c1a95bc438 Nov 01 22:40:32.556945 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:32.556917 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b0f6226e-58c9f9b8f9-nvhd6"] Nov 01 22:40:32.582998 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:32.582965 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b0f6226e-69867d597-9crx9" event={"ID":"84e79b24-6ac7-474d-bb27-ec1ce218afbb","Type":"ContainerStarted","Data":"b3e918099be1811117478d5f93cdf1331abad2767115ae0ecbe494c1a95bc438"} Nov 01 22:40:32.584491 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:32.584464 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0f6226e-58c9f9b8f9-nvhd6" event={"ID":"baa15442-fd2c-4bdb-90c8-ff3c043ed73e","Type":"ContainerStarted","Data":"17a939621c22a1bb8fe7a72f6e7fd8d120d1923b6f6802426db54d807b258bf7"} Nov 01 22:40:33.595699 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:33.595657 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b0f6226e-69867d597-9crx9" event={"ID":"84e79b24-6ac7-474d-bb27-ec1ce218afbb","Type":"ContainerStarted","Data":"1c8cc0f620d9ac4e09aec037d80c64bd926961aef90df6217b7a120d619ac633"} Nov 01 22:40:33.596290 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:33.595747 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b0f6226e-69867d597-9crx9" Nov 01 22:40:33.598162 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:33.598134 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0f6226e-58c9f9b8f9-nvhd6" event={"ID":"baa15442-fd2c-4bdb-90c8-ff3c043ed73e","Type":"ContainerStarted","Data":"d0d09aa4410e55d94e499daf5eadd71636f5b70622183ced7bf1c32f1e50e32c"} Nov 01 22:40:33.598894 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:33.598874 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0f6226e-58c9f9b8f9-nvhd6" Nov 01 22:40:33.629094 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:33.629038 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b0f6226e-69867d597-9crx9" podStartSLOduration=2.629023286 podStartE2EDuration="2.629023286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:33.625641887 +0000 UTC m=+1002.030232520" watchObservedRunningTime="2026-11-01 22:40:33.629023286 +0000 UTC m=+1002.033613949" Nov 01 22:40:33.653435 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:33.653365 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0f6226e-58c9f9b8f9-nvhd6" podStartSLOduration=2.653348248 podStartE2EDuration="2.653348248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:33.652709036 +0000 UTC m=+1002.057299692" watchObservedRunningTime="2026-11-01 22:40:33.653348248 +0000 UTC m=+1002.057938884" Nov 01 22:40:36.017815 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:36.017776 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e61a2d4e-7c6d5f9d55-59nj8"] Nov 01 22:40:36.023225 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:36.023197 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e61a2d4e-7c6d5f9d55-59nj8" Nov 01 22:40:36.028098 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:36.028066 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-e61a2d4e\"" Nov 01 22:40:36.041296 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:36.041264 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-e61a2d4e-dockercfg-kphqs\"" Nov 01 22:40:36.048739 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:36.048703 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e61a2d4e-7c6d5f9d55-59nj8"] Nov 01 22:40:36.139858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:36.138543 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88140aed-a686-45ba-b920-465c9aef3b53-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e61a2d4e-7c6d5f9d55-59nj8\" (UID: \"88140aed-a686-45ba-b920-465c9aef3b53\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e61a2d4e-7c6d5f9d55-59nj8" Nov 01 22:40:36.139858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:36.138698 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-855jr\" (UniqueName: \"kubernetes.io/projected/88140aed-a686-45ba-b920-465c9aef3b53-kube-api-access-855jr\") pod \"payload-pre-processing-e2e-derive-e61a2d4e-7c6d5f9d55-59nj8\" (UID: \"88140aed-a686-45ba-b920-465c9aef3b53\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e61a2d4e-7c6d5f9d55-59nj8" Nov 01 22:40:36.248861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:36.243114 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88140aed-a686-45ba-b920-465c9aef3b53-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e61a2d4e-7c6d5f9d55-59nj8\" (UID: \"88140aed-a686-45ba-b920-465c9aef3b53\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e61a2d4e-7c6d5f9d55-59nj8" Nov 01 22:40:36.248861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:36.243188 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-855jr\" (UniqueName: \"kubernetes.io/projected/88140aed-a686-45ba-b920-465c9aef3b53-kube-api-access-855jr\") pod \"payload-pre-processing-e2e-derive-e61a2d4e-7c6d5f9d55-59nj8\" (UID: \"88140aed-a686-45ba-b920-465c9aef3b53\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e61a2d4e-7c6d5f9d55-59nj8" Nov 01 22:40:36.248861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:36.244779 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88140aed-a686-45ba-b920-465c9aef3b53-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e61a2d4e-7c6d5f9d55-59nj8\" (UID: \"88140aed-a686-45ba-b920-465c9aef3b53\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e61a2d4e-7c6d5f9d55-59nj8" Nov 01 22:40:36.305905 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:36.304656 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-855jr\" (UniqueName: \"kubernetes.io/projected/88140aed-a686-45ba-b920-465c9aef3b53-kube-api-access-855jr\") pod \"payload-pre-processing-e2e-derive-e61a2d4e-7c6d5f9d55-59nj8\" (UID: \"88140aed-a686-45ba-b920-465c9aef3b53\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e61a2d4e-7c6d5f9d55-59nj8" Nov 01 22:40:36.348861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:36.346132 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e61a2d4e-7c6d5f9d55-59nj8" Nov 01 22:40:36.889963 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:36.883730 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e61a2d4e-7c6d5f9d55-59nj8"] Nov 01 22:40:37.645128 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:37.645095 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e61a2d4e-7c6d5f9d55-59nj8" event={"ID":"88140aed-a686-45ba-b920-465c9aef3b53","Type":"ContainerStarted","Data":"a639e9cab64c7ee3f3b79493bd1ac74334ab79d5a0e2af60abdf017bb1a06ad9"} Nov 01 22:40:37.645536 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:37.645155 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e61a2d4e-7c6d5f9d55-59nj8" event={"ID":"88140aed-a686-45ba-b920-465c9aef3b53","Type":"ContainerStarted","Data":"12bf2c56e3fdef6350e9d3669e0c9c302d0558164af6621121088e2a3ce5632f"} Nov 01 22:40:37.645536 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:37.645182 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-e61a2d4e-7c6d5f9d55-59nj8" Nov 01 22:40:40.330695 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:40.330632 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-e61a2d4e-7c6d5f9d55-59nj8" podStartSLOduration=5.330612773 podStartE2EDuration="5.330612773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:37.676130007 +0000 UTC m=+1006.080720669" watchObservedRunningTime="2026-11-01 22:40:40.330612773 +0000 UTC m=+1008.735203403" Nov 01 22:40:40.334944 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:40.334789 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47"] Nov 01 22:40:40.340555 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:40.340519 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47" Nov 01 22:40:40.349258 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:40.344950 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b0f6226e-serving-cert\"" Nov 01 22:40:40.352943 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:40.352819 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47"] Nov 01 22:40:40.411299 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:40.411263 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tptlf\" (UniqueName: \"kubernetes.io/projected/17c17145-3afc-4f03-a7a9-6a344420af10-kube-api-access-tptlf\") pod \"maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47\" (UID: \"17c17145-3afc-4f03-a7a9-6a344420af10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47" Nov 01 22:40:40.411476 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:40.411334 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17c17145-3afc-4f03-a7a9-6a344420af10-maas-api-tls\") pod \"maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47\" (UID: \"17c17145-3afc-4f03-a7a9-6a344420af10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47" Nov 01 22:40:40.411574 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:40.411548 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17c17145-3afc-4f03-a7a9-6a344420af10-openshift-service-ca\") pod \"maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47\" (UID: \"17c17145-3afc-4f03-a7a9-6a344420af10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47" Nov 01 22:40:40.411681 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:40.411661 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17c17145-3afc-4f03-a7a9-6a344420af10-metrics-certs\") pod \"maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47\" (UID: \"17c17145-3afc-4f03-a7a9-6a344420af10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47" Nov 01 22:40:40.516861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:40.515944 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17c17145-3afc-4f03-a7a9-6a344420af10-metrics-certs\") pod \"maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47\" (UID: \"17c17145-3afc-4f03-a7a9-6a344420af10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47" Nov 01 22:40:40.516861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:40.516113 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tptlf\" (UniqueName: \"kubernetes.io/projected/17c17145-3afc-4f03-a7a9-6a344420af10-kube-api-access-tptlf\") pod \"maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47\" (UID: \"17c17145-3afc-4f03-a7a9-6a344420af10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47" Nov 01 22:40:40.516861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:40.516182 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17c17145-3afc-4f03-a7a9-6a344420af10-maas-api-tls\") pod \"maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47\" (UID: \"17c17145-3afc-4f03-a7a9-6a344420af10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47" Nov 01 22:40:40.516861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:40.516417 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17c17145-3afc-4f03-a7a9-6a344420af10-openshift-service-ca\") pod \"maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47\" (UID: \"17c17145-3afc-4f03-a7a9-6a344420af10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47" Nov 01 22:40:40.524512 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:40.517272 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17c17145-3afc-4f03-a7a9-6a344420af10-openshift-service-ca\") pod \"maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47\" (UID: \"17c17145-3afc-4f03-a7a9-6a344420af10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47" Nov 01 22:40:40.533645 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:40.525912 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17c17145-3afc-4f03-a7a9-6a344420af10-metrics-certs\") pod \"maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47\" (UID: \"17c17145-3afc-4f03-a7a9-6a344420af10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47" Nov 01 22:40:40.544854 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:40.540792 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17c17145-3afc-4f03-a7a9-6a344420af10-maas-api-tls\") pod \"maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47\" (UID: \"17c17145-3afc-4f03-a7a9-6a344420af10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47" Nov 01 22:40:40.544854 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:40.543307 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tptlf\" (UniqueName: \"kubernetes.io/projected/17c17145-3afc-4f03-a7a9-6a344420af10-kube-api-access-tptlf\") pod \"maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47\" (UID: \"17c17145-3afc-4f03-a7a9-6a344420af10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47" Nov 01 22:40:40.663121 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:40.663032 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47" Nov 01 22:40:40.997563 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:40.996465 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47"] Nov 01 22:40:41.125218 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:41.125189 5011 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:40:41.675782 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:41.675749 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47" event={"ID":"17c17145-3afc-4f03-a7a9-6a344420af10","Type":"ContainerStarted","Data":"cd1dfe6c59773001071f5418234eba3959cba6cf7e5bb94abcbb979104d75594"} Nov 01 22:40:42.590292 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:42.590260 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-88896ec5-677cd44fc5-xf2wh" Nov 01 22:40:42.686573 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:42.686527 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47" event={"ID":"17c17145-3afc-4f03-a7a9-6a344420af10","Type":"ContainerStarted","Data":"34d9926e54d855f994f2676ecd7499ccb44fe16f4b3371a8d80b09666a9d68b1"} Nov 01 22:40:42.687109 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:42.686636 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47" Nov 01 22:40:42.710177 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:42.710121 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47" podStartSLOduration=2.108071922 podStartE2EDuration="2.710106337s" totalImagesPullingTime="602.034415ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:42.706851696 +0000 UTC m=+1011.111442331" watchObservedRunningTime="2026-11-01 22:40:42.710106337 +0000 UTC m=+1011.114696968" Nov 01 22:40:44.650861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:44.648030 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b0f6226e-69867d597-9crx9" Nov 01 22:40:45.632444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:45.632415 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0f6226e-58c9f9b8f9-nvhd6" Nov 01 22:40:47.282879 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:40:47.281135 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:40:47.427966 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:47.426934 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:47.949859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:47.949106 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:48.152732 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:48.145097 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:48.171361 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:48.168931 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:48.201083 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:48.200979 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:40:48.303864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:48.303192 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:48.303864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:48.303504 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 01 22:40:48.322862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:48.322239 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:48.434978 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:48.434927 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:48.435883 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:48.435854 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:40:48.462928 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:48.462800 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:48.666859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:48.615205 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:48.806245 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:48.668043 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 22:40:48.957731 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:48.951680 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-e61a2d4e-7c6d5f9d55-59nj8" Nov 01 22:40:49.018695 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:49.015017 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47" Nov 01 22:40:49.177408 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:49.170932 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:49.177408 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:49.172491 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:40:49.273865 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:49.272054 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:40:49.373860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:49.274117 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:40:49.373860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:49.319866 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:40:49.373860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:49.331346 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 01 22:40:49.373860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:49.331804 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:40:49.373860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:49.331891 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:40:49.478246 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:49.472124 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:40:49.513983 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:49.479985 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 01 22:40:49.524784 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:49.493864 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:49.526322 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:49.526287 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:40:49.526469 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:49.526367 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:40:49.546175 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:49.493980 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:40:49.546175 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:49.527409 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" podUID="5913d691-0d87-4599-b176-d132888e187e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:49.811912 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:49.784774 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:40:49.857859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:49.814583 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:40:50.289857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:50.289424 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:40:50.306859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:50.290107 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded" Nov 01 22:40:50.306859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:50.303322 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:50.421857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:50.421087 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:40:50.457864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:50.457314 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:50.458079 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:50.457932 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 01 22:40:50.715861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:50.701934 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:50.715861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:50.712706 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:50.775860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:50.774970 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:50.878869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:50.875738 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:40:51.036034 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:51.032941 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:40:51.036034 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:51.033032 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" podUID="db213002-5fde-4c7a-ba7e-6762c011b4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:51.124263 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:40:51.100095 5011 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.035s" Nov 01 22:40:51.124263 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:51.100181 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:40:51.333915 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:51.304007 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:40:51.333915 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:51.305009 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:51.488734 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:51.488691 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:51.489765 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:51.489718 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:40:51.498039 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:51.497929 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:40:51.845859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:51.820459 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": context deadline exceeded" Nov 01 22:40:51.937873 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:51.923205 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:51.986859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:51.740812 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:51.986859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:51.964635 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:40:51.986859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:51.965474 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 01 22:40:52.324286 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:52.324232 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-474rg" podUID="72f1a530-bd82-48d6-a9a7-020eacd1b1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:52.473854 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:52.229921 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7q99w" podUID="643613a9-01d4-4d8e-bfa3-3cc8b7e118b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Nov 01 22:40:52.949520 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:52.782218 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:40:52.949520 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:52.782973 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:40:52.949520 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:52.864243 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:40:52.949520 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:52.880463 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:52.949520 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:52.880971 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:52.992033 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:52.991977 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:40:53.052544 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:53.052494 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:53.054953 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:53.037554 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:53.147154 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:40:53.147112 5011 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.123s" Nov 01 22:40:53.147624 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:53.147598 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:40:53.147787 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:53.147768 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:40:53.147927 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:53.147906 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:40:53.148162 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:53.148152 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:40:53.149116 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:53.149087 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:40:53.149224 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:53.149157 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:40:53.149224 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:53.149218 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:40:53.155355 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:53.155323 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:40:53.157335 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:53.157158 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:40:56.341933 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:56.341901 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b"] Nov 01 22:40:56.349517 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:56.349486 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" Nov 01 22:40:56.369231 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:56.369202 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b"] Nov 01 22:40:56.445291 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:56.445254 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hsr7b\" (UID: \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" Nov 01 22:40:56.445480 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:56.445366 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wpk2\" (UniqueName: \"kubernetes.io/projected/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-kube-api-access-6wpk2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hsr7b\" (UID: \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" Nov 01 22:40:56.445480 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:56.445447 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hsr7b\" (UID: \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" Nov 01 22:40:56.445572 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:56.445551 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hsr7b\" (UID: \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" Nov 01 22:40:56.445611 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:56.445595 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hsr7b\" (UID: \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" Nov 01 22:40:56.445792 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:56.445769 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hsr7b\" (UID: \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" Nov 01 22:40:56.445942 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:56.445926 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hsr7b\" (UID: \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" Nov 01 22:40:56.547610 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:56.547573 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hsr7b\" (UID: \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" Nov 01 22:40:56.547811 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:56.547636 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wpk2\" (UniqueName: \"kubernetes.io/projected/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-kube-api-access-6wpk2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hsr7b\" (UID: \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" Nov 01 22:40:56.547811 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:56.547690 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hsr7b\" (UID: \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" Nov 01 22:40:56.547811 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:56.547735 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hsr7b\" (UID: \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" Nov 01 22:40:56.547811 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:56.547784 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hsr7b\" (UID: \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" Nov 01 22:40:56.548068 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:56.547917 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hsr7b\" (UID: \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" Nov 01 22:40:56.548068 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:56.548000 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hsr7b\" (UID: \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" Nov 01 22:40:56.548702 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:56.548604 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hsr7b\" (UID: \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" Nov 01 22:40:56.550433 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:56.550408 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hsr7b\" (UID: \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" Nov 01 22:40:56.550536 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:56.550445 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hsr7b\" (UID: \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" Nov 01 22:40:56.550674 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:56.550643 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hsr7b\" (UID: \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" Nov 01 22:40:56.550889 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:56.550870 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hsr7b\" (UID: \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" Nov 01 22:40:56.556413 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:56.556394 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wpk2\" (UniqueName: \"kubernetes.io/projected/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-kube-api-access-6wpk2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hsr7b\" (UID: \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" Nov 01 22:40:56.556523 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:56.556507 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hsr7b\" (UID: \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" Nov 01 22:40:56.672080 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:56.672052 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" Nov 01 22:40:57.031333 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:57.031019 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b"] Nov 01 22:40:57.079204 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:57.079139 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:40:57.079406 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:57.079249 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:40:57.178375 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:57.178332 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" event={"ID":"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3","Type":"ContainerStarted","Data":"667350b76a7205dd92fdedba72efd0cc783167d285efad3d3707d4d68195b324"} Nov 01 22:40:57.178574 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:57.178399 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" event={"ID":"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3","Type":"ContainerStarted","Data":"a19cecfb1a0fbf8e2afb73ffc409f9bfa3c16b056112a93fbf414db4eea03ac1"} Nov 01 22:40:57.178574 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:57.178476 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" Nov 01 22:40:57.203467 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:57.203405 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" podStartSLOduration=1.203385695 podStartE2EDuration="1.203385695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:57.199264977 +0000 UTC m=+1025.603855610" watchObservedRunningTime="2026-11-01 22:40:57.203385695 +0000 UTC m=+1025.607976328" Nov 01 22:40:58.189068 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:58.189034 5011 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-hsr7b container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:40:58.189455 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:40:58.189096 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" podUID="00f9ac03-3cbe-4d33-9a63-feb3099cb6d3" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:41:01.188493 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:01.188445 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" Nov 01 22:41:08.759198 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:08.759161 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-88896ec5-677cd44fc5-xf2wh"] Nov 01 22:41:11.290044 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:11.290005 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b0f6226e-69867d597-9crx9"] Nov 01 22:41:11.302876 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:11.302810 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b0f6226e-58c9f9b8f9-nvhd6"] Nov 01 22:41:13.760448 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:13.760408 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-88896ec5-677cd44fc5-xf2wh" podUID="f0215a01-c33e-4921-b292-88ad9d3dbb31" containerName="payload-pre-processing" containerID="cri-o://294c9b5699db937f3ce89646ac91e1fab43246ffece420ef8708c76265796893" gracePeriod=25 Nov 01 22:41:14.071398 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:14.070913 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-88896ec5-677cd44fc5-xf2wh" Nov 01 22:41:14.108076 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:14.108023 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0215a01-c33e-4921-b292-88ad9d3dbb31-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0215a01-c33e-4921-b292-88ad9d3dbb31-plugins-config-volume\") pod \"f0215a01-c33e-4921-b292-88ad9d3dbb31\" (UID: \"f0215a01-c33e-4921-b292-88ad9d3dbb31\") " Nov 01 22:41:14.108476 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:14.108446 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0215a01-c33e-4921-b292-88ad9d3dbb31-kube-api-access-j7ncl\" (UniqueName: \"kubernetes.io/projected/f0215a01-c33e-4921-b292-88ad9d3dbb31-kube-api-access-j7ncl\") pod \"f0215a01-c33e-4921-b292-88ad9d3dbb31\" (UID: \"f0215a01-c33e-4921-b292-88ad9d3dbb31\") " Nov 01 22:41:14.108662 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:14.108447 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0215a01-c33e-4921-b292-88ad9d3dbb31-plugins-config-volume" pod "f0215a01-c33e-4921-b292-88ad9d3dbb31" (UID: "f0215a01-c33e-4921-b292-88ad9d3dbb31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:41:14.109151 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:14.109130 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0215a01-c33e-4921-b292-88ad9d3dbb31-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:14.112784 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:14.112166 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0215a01-c33e-4921-b292-88ad9d3dbb31-kube-api-access-j7ncl" pod "f0215a01-c33e-4921-b292-88ad9d3dbb31" (UID: "f0215a01-c33e-4921-b292-88ad9d3dbb31"). InnerVolumeSpecName "kube-api-access-j7ncl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:41:14.211690 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:14.211650 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7ncl\" (UniqueName: \"kubernetes.io/projected/f0215a01-c33e-4921-b292-88ad9d3dbb31-kube-api-access-j7ncl\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:14.749855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:14.749510 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c"] Nov 01 22:41:14.749855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:14.749747 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" podUID="20b6cb03-c7e7-478e-ba20-02a34c02d602" containerName="istio-proxy" containerID="cri-o://81f9c40f2e3851ba5d6dfad1138b53d08cb95fcae97e665ee1d2fc6901b58797" gracePeriod=30 Nov 01 22:41:14.795087 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:14.784520 5011 generic.go:356] "Generic (PLEG): container finished" podID="f0215a01-c33e-4921-b292-88ad9d3dbb31" containerID="294c9b5699db937f3ce89646ac91e1fab43246ffece420ef8708c76265796893" exitCode=0 Nov 01 22:41:14.795087 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:14.784574 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-88896ec5-677cd44fc5-xf2wh" event={"ID":"f0215a01-c33e-4921-b292-88ad9d3dbb31","Type":"ContainerDied","Data":"294c9b5699db937f3ce89646ac91e1fab43246ffece420ef8708c76265796893"} Nov 01 22:41:14.795087 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:14.784624 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-88896ec5-677cd44fc5-xf2wh" event={"ID":"f0215a01-c33e-4921-b292-88ad9d3dbb31","Type":"ContainerDied","Data":"ab8f85913fe5da086c864d6f8e47f1565720a2791873e6efbe0b2d5c552d3dd7"} Nov 01 22:41:14.795087 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:14.784652 5011 scope.go:122] "RemoveContainer" containerID="294c9b5699db937f3ce89646ac91e1fab43246ffece420ef8708c76265796893" Nov 01 22:41:14.795087 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:14.784816 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-88896ec5-677cd44fc5-xf2wh" Nov 01 22:41:15.115857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:15.110158 5011 scope.go:122] "RemoveContainer" containerID="294c9b5699db937f3ce89646ac91e1fab43246ffece420ef8708c76265796893" Nov 01 22:41:15.138854 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:41:15.136364 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"294c9b5699db937f3ce89646ac91e1fab43246ffece420ef8708c76265796893\": container with ID starting with 294c9b5699db937f3ce89646ac91e1fab43246ffece420ef8708c76265796893 not found: ID does not exist" containerID="294c9b5699db937f3ce89646ac91e1fab43246ffece420ef8708c76265796893" Nov 01 22:41:15.138854 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:15.136413 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"294c9b5699db937f3ce89646ac91e1fab43246ffece420ef8708c76265796893"} err="failed to get container status \"294c9b5699db937f3ce89646ac91e1fab43246ffece420ef8708c76265796893\": rpc error: code = NotFound desc = could not find container \"294c9b5699db937f3ce89646ac91e1fab43246ffece420ef8708c76265796893\": container with ID starting with 294c9b5699db937f3ce89646ac91e1fab43246ffece420ef8708c76265796893 not found: ID does not exist" Nov 01 22:41:15.352928 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:15.333534 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:41:15.403940 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:15.403905 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e61a2d4e-7c6d5f9d55-59nj8"] Nov 01 22:41:15.480790 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:15.480756 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-88896ec5-677cd44fc5-xf2wh"] Nov 01 22:41:16.023883 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:16.021850 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-88896ec5-677cd44fc5-xf2wh"] Nov 01 22:41:16.291028 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:16.290918 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b0f6226e-69867d597-9crx9" podUID="84e79b24-6ac7-474d-bb27-ec1ce218afbb" containerName="payload-processing" containerID="cri-o://1c8cc0f620d9ac4e09aec037d80c64bd926961aef90df6217b7a120d619ac633" gracePeriod=25 Nov 01 22:41:16.304059 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:16.303979 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0f6226e-58c9f9b8f9-nvhd6" podUID="baa15442-fd2c-4bdb-90c8-ff3c043ed73e" containerName="payload-pre-processing" containerID="cri-o://d0d09aa4410e55d94e499daf5eadd71636f5b70622183ced7bf1c32f1e50e32c" gracePeriod=25 Nov 01 22:41:17.230077 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:17.230043 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b0f6226e-69867d597-9crx9" Nov 01 22:41:17.498430 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:17.476662 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84e79b24-6ac7-474d-bb27-ec1ce218afbb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e79b24-6ac7-474d-bb27-ec1ce218afbb-plugins-config-volume\") pod \"84e79b24-6ac7-474d-bb27-ec1ce218afbb\" (UID: \"84e79b24-6ac7-474d-bb27-ec1ce218afbb\") " Nov 01 22:41:17.499155 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:17.499124 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84e79b24-6ac7-474d-bb27-ec1ce218afbb-kube-api-access-65qcc\" (UniqueName: \"kubernetes.io/projected/84e79b24-6ac7-474d-bb27-ec1ce218afbb-kube-api-access-65qcc\") pod \"84e79b24-6ac7-474d-bb27-ec1ce218afbb\" (UID: \"84e79b24-6ac7-474d-bb27-ec1ce218afbb\") " Nov 01 22:41:17.501375 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:17.501347 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e79b24-6ac7-474d-bb27-ec1ce218afbb-plugins-config-volume" pod "84e79b24-6ac7-474d-bb27-ec1ce218afbb" (UID: "84e79b24-6ac7-474d-bb27-ec1ce218afbb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:41:17.545703 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:17.545628 5011 generic.go:356] "Generic (PLEG): container finished" podID="baa15442-fd2c-4bdb-90c8-ff3c043ed73e" containerID="d0d09aa4410e55d94e499daf5eadd71636f5b70622183ced7bf1c32f1e50e32c" exitCode=0 Nov 01 22:41:17.553067 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:17.553012 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0f6226e-58c9f9b8f9-nvhd6" event={"ID":"baa15442-fd2c-4bdb-90c8-ff3c043ed73e","Type":"ContainerDied","Data":"d0d09aa4410e55d94e499daf5eadd71636f5b70622183ced7bf1c32f1e50e32c"} Nov 01 22:41:17.600464 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:17.600422 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e79b24-6ac7-474d-bb27-ec1ce218afbb-kube-api-access-65qcc" pod "84e79b24-6ac7-474d-bb27-ec1ce218afbb" (UID: "84e79b24-6ac7-474d-bb27-ec1ce218afbb"). InnerVolumeSpecName "kube-api-access-65qcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:41:17.650905 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:17.610818 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e79b24-6ac7-474d-bb27-ec1ce218afbb-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:17.650905 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:17.610875 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65qcc\" (UniqueName: \"kubernetes.io/projected/84e79b24-6ac7-474d-bb27-ec1ce218afbb-kube-api-access-65qcc\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:17.703356 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:17.703316 5011 generic.go:356] "Generic (PLEG): container finished" podID="84e79b24-6ac7-474d-bb27-ec1ce218afbb" containerID="1c8cc0f620d9ac4e09aec037d80c64bd926961aef90df6217b7a120d619ac633" exitCode=0 Nov 01 22:41:17.703664 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:17.703641 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b0f6226e-69867d597-9crx9" event={"ID":"84e79b24-6ac7-474d-bb27-ec1ce218afbb","Type":"ContainerDied","Data":"1c8cc0f620d9ac4e09aec037d80c64bd926961aef90df6217b7a120d619ac633"} Nov 01 22:41:17.703823 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:17.703801 5011 scope.go:122] "RemoveContainer" containerID="1c8cc0f620d9ac4e09aec037d80c64bd926961aef90df6217b7a120d619ac633" Nov 01 22:41:17.704073 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:17.704054 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b0f6226e-69867d597-9crx9" Nov 01 22:41:17.776902 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:17.776570 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:41:17.813856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:17.811120 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 01 22:41:17.875894 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:17.856974 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:17.883856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:17.882990 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:17.944859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:17.937245 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:17.958865 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:17.954812 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0f6226e-58c9f9b8f9-nvhd6" Nov 01 22:41:18.040323 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:18.040238 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0215a01-c33e-4921-b292-88ad9d3dbb31" path="/var/lib/kubelet/pods/f0215a01-c33e-4921-b292-88ad9d3dbb31/volumes" Nov 01 22:41:18.242507 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:18.241633 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded" Nov 01 22:41:18.270116 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:18.261303 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baa15442-fd2c-4bdb-90c8-ff3c043ed73e-kube-api-access-pdkvb\" (UniqueName: \"kubernetes.io/projected/baa15442-fd2c-4bdb-90c8-ff3c043ed73e-kube-api-access-pdkvb\") pod \"baa15442-fd2c-4bdb-90c8-ff3c043ed73e\" (UID: \"baa15442-fd2c-4bdb-90c8-ff3c043ed73e\") " Nov 01 22:41:18.270116 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:18.261423 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baa15442-fd2c-4bdb-90c8-ff3c043ed73e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa15442-fd2c-4bdb-90c8-ff3c043ed73e-plugins-config-volume\") pod \"baa15442-fd2c-4bdb-90c8-ff3c043ed73e\" (UID: \"baa15442-fd2c-4bdb-90c8-ff3c043ed73e\") " Nov 01 22:41:18.281949 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:18.279531 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baa15442-fd2c-4bdb-90c8-ff3c043ed73e-plugins-config-volume" pod "baa15442-fd2c-4bdb-90c8-ff3c043ed73e" (UID: "baa15442-fd2c-4bdb-90c8-ff3c043ed73e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:41:18.343874 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:18.343286 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:18.345635 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:18.344465 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 22:41:18.372327 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:18.369690 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa15442-fd2c-4bdb-90c8-ff3c043ed73e-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:18.402436 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:18.402390 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baa15442-fd2c-4bdb-90c8-ff3c043ed73e-kube-api-access-pdkvb" pod "baa15442-fd2c-4bdb-90c8-ff3c043ed73e" (UID: "baa15442-fd2c-4bdb-90c8-ff3c043ed73e"). InnerVolumeSpecName "kube-api-access-pdkvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:41:18.471876 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:18.471347 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdkvb\" (UniqueName: \"kubernetes.io/projected/baa15442-fd2c-4bdb-90c8-ff3c043ed73e-kube-api-access-pdkvb\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:18.496488 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:18.496020 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:18.510969 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:18.510924 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:18.567244 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:18.546996 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:18.609985 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:18.609888 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b0f6226e-69867d597-9crx9"] Nov 01 22:41:18.676255 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:41:18.676204 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:41:18.722884 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:18.722847 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b0f6226e-69867d597-9crx9"] Nov 01 22:41:18.760857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:18.759779 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0f6226e-58c9f9b8f9-nvhd6" event={"ID":"baa15442-fd2c-4bdb-90c8-ff3c043ed73e","Type":"ContainerDied","Data":"17a939621c22a1bb8fe7a72f6e7fd8d120d1923b6f6802426db54d807b258bf7"} Nov 01 22:41:18.760857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:18.759871 5011 scope.go:122] "RemoveContainer" containerID="d0d09aa4410e55d94e499daf5eadd71636f5b70622183ced7bf1c32f1e50e32c" Nov 01 22:41:18.760857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:18.759973 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0f6226e-58c9f9b8f9-nvhd6" Nov 01 22:41:18.796859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:18.784972 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:18.796859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:18.785597 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:41:18.813487 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:18.813436 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 01 22:41:18.821054 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:18.821028 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:41:18.958861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:18.948330 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:18.958861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:18.949007 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:41:18.958861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:18.954168 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 01 22:41:18.958861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:18.954725 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:41:19.306916 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:19.239711 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b0f6226e-58c9f9b8f9-nvhd6"] Nov 01 22:41:19.527818 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:19.523061 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:19.527818 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:19.523972 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:41:19.527818 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:19.526296 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:41:19.527818 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:19.526338 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" podUID="5913d691-0d87-4599-b176-d132888e187e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:19.527818 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:19.527458 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:41:19.551860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:19.550143 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:41:19.576927 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:19.558759 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 22:41:19.621086 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:19.566172 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:41:19.621086 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:19.620377 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:41:19.621086 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:19.620460 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:41:19.725865 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:19.586314 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 01 22:41:19.735920 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:19.640075 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:19.735920 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:19.728375 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:41:19.795207 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:19.795153 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:19.819445 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:19.819388 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 01 22:41:20.158272 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:20.025985 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:41:20.313055 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:20.257069 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b0f6226e-58c9f9b8f9-nvhd6"] Nov 01 22:41:20.504917 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:20.501078 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-e61a2d4e-7c6d5f9d55-59nj8" podUID="88140aed-a686-45ba-b920-465c9aef3b53" containerName="payload-pre-processing" containerID="cri-o://a639e9cab64c7ee3f3b79493bd1ac74334ab79d5a0e2af60abdf017bb1a06ad9" gracePeriod=25 Nov 01 22:41:20.648742 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:20.648685 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 01 22:41:20.735356 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:20.735285 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:41:20.781062 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:20.780949 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:20.997339 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:20.994715 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:41:21.020858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:20.998122 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:41:21.020858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:20.998422 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 22:41:21.020858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:20.998617 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:41:21.020858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:20.999690 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:41:21.020858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:20.999729 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" podUID="db213002-5fde-4c7a-ba7e-6762c011b4a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:21.117908 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:21.109972 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 01 22:41:21.141202 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:21.110149 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:41:21.492854 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:21.459147 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84e79b24-6ac7-474d-bb27-ec1ce218afbb" path="/var/lib/kubelet/pods/84e79b24-6ac7-474d-bb27-ec1ce218afbb/volumes" Nov 01 22:41:21.494250 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:21.494219 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baa15442-fd2c-4bdb-90c8-ff3c043ed73e" path="/var/lib/kubelet/pods/baa15442-fd2c-4bdb-90c8-ff3c043ed73e/volumes" Nov 01 22:41:21.547879 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:41:21.542483 5011 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.225s" Nov 01 22:41:21.547879 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:21.542636 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47"] Nov 01 22:41:21.550648 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:21.550612 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47" podUID="17c17145-3afc-4f03-a7a9-6a344420af10" containerName="maas-api" containerID="cri-o://34d9926e54d855f994f2676ecd7499ccb44fe16f4b3371a8d80b09666a9d68b1" gracePeriod=30 Nov 01 22:41:21.667516 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:21.653864 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:21.762071 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:21.762015 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:21.763279 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:21.763230 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:21.820258 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:21.801093 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:22.005350 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:22.005307 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:22.006096 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:22.006062 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:22.006529 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:22.006505 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 01 22:41:22.006929 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:22.006905 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 22:41:22.126856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:22.124930 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:22.165770 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:22.165721 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:41:22.679796 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:22.679742 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 01 22:41:22.829611 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:22.781267 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:22.843276 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:22.837513 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:22.948864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:22.947287 5011 generic.go:356] "Generic (PLEG): container finished" podID="88140aed-a686-45ba-b920-465c9aef3b53" containerID="a639e9cab64c7ee3f3b79493bd1ac74334ab79d5a0e2af60abdf017bb1a06ad9" exitCode=0 Nov 01 22:41:22.948864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:22.947526 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e61a2d4e-7c6d5f9d55-59nj8" event={"ID":"88140aed-a686-45ba-b920-465c9aef3b53","Type":"ContainerDied","Data":"a639e9cab64c7ee3f3b79493bd1ac74334ab79d5a0e2af60abdf017bb1a06ad9"} Nov 01 22:41:22.999396 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:22.987472 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 22:41:23.006005 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.005969 5011 generic.go:356] "Generic (PLEG): container finished" podID="17c17145-3afc-4f03-a7a9-6a344420af10" containerID="34d9926e54d855f994f2676ecd7499ccb44fe16f4b3371a8d80b09666a9d68b1" exitCode=0 Nov 01 22:41:23.006405 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.006380 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47" event={"ID":"17c17145-3afc-4f03-a7a9-6a344420af10","Type":"ContainerDied","Data":"34d9926e54d855f994f2676ecd7499ccb44fe16f4b3371a8d80b09666a9d68b1"} Nov 01 22:41:23.085947 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.020601 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:23.085947 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.035549 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:23.085947 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.035868 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:41:23.085947 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.036098 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:41:23.139143 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.131212 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 22:41:23.139143 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.132250 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:23.656713 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.656164 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:41:23.656713 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.656230 5011 generic.go:356] "Generic (PLEG): container finished" podID="20b6cb03-c7e7-478e-ba20-02a34c02d602" containerID="81f9c40f2e3851ba5d6dfad1138b53d08cb95fcae97e665ee1d2fc6901b58797" exitCode=0 Nov 01 22:41:23.656713 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.656269 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" event={"ID":"20b6cb03-c7e7-478e-ba20-02a34c02d602","Type":"ContainerDied","Data":"81f9c40f2e3851ba5d6dfad1138b53d08cb95fcae97e665ee1d2fc6901b58797"} Nov 01 22:41:23.682482 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.682457 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 22:41:23.710004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.709973 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 22:41:23.774102 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.774059 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 22:41:23.779137 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.779096 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47" Nov 01 22:41:23.786711 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.784461 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:41:23.830551 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.830477 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:41:23.879812 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.867968 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 22:41:23.879812 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.869205 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 22:41:23.923708 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.920622 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20b6cb03-c7e7-478e-ba20-02a34c02d602-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b6cb03-c7e7-478e-ba20-02a34c02d602-istiod-ca-cert\") pod \"20b6cb03-c7e7-478e-ba20-02a34c02d602\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " Nov 01 22:41:23.923708 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.922029 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17c17145-3afc-4f03-a7a9-6a344420af10-kube-api-access-tptlf\" (UniqueName: \"kubernetes.io/projected/17c17145-3afc-4f03-a7a9-6a344420af10-kube-api-access-tptlf\") pod \"17c17145-3afc-4f03-a7a9-6a344420af10\" (UID: \"17c17145-3afc-4f03-a7a9-6a344420af10\") " Nov 01 22:41:23.923708 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.922082 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20b6cb03-c7e7-478e-ba20-02a34c02d602-kube-api-access-pqwc4\" (UniqueName: \"kubernetes.io/projected/20b6cb03-c7e7-478e-ba20-02a34c02d602-kube-api-access-pqwc4\") pod \"20b6cb03-c7e7-478e-ba20-02a34c02d602\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " Nov 01 22:41:23.923708 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.922120 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20b6cb03-c7e7-478e-ba20-02a34c02d602-istio-token\" (UniqueName: \"kubernetes.io/projected/20b6cb03-c7e7-478e-ba20-02a34c02d602-istio-token\") pod \"20b6cb03-c7e7-478e-ba20-02a34c02d602\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " Nov 01 22:41:23.923708 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.922212 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17c17145-3afc-4f03-a7a9-6a344420af10-metrics-certs\" (UniqueName: \"kubernetes.io/secret/17c17145-3afc-4f03-a7a9-6a344420af10-metrics-certs\") pod \"17c17145-3afc-4f03-a7a9-6a344420af10\" (UID: \"17c17145-3afc-4f03-a7a9-6a344420af10\") " Nov 01 22:41:23.923708 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.922264 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-workload-socket\") pod \"20b6cb03-c7e7-478e-ba20-02a34c02d602\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " Nov 01 22:41:23.923708 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.922327 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-istio-envoy\") pod \"20b6cb03-c7e7-478e-ba20-02a34c02d602\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " Nov 01 22:41:23.923708 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.922408 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20b6cb03-c7e7-478e-ba20-02a34c02d602-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b6cb03-c7e7-478e-ba20-02a34c02d602-istio-podinfo\") pod \"20b6cb03-c7e7-478e-ba20-02a34c02d602\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " Nov 01 22:41:23.923708 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.922453 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17c17145-3afc-4f03-a7a9-6a344420af10-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17c17145-3afc-4f03-a7a9-6a344420af10-maas-api-tls\") pod \"17c17145-3afc-4f03-a7a9-6a344420af10\" (UID: \"17c17145-3afc-4f03-a7a9-6a344420af10\") " Nov 01 22:41:23.923708 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.922496 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-workload-certs\") pod \"20b6cb03-c7e7-478e-ba20-02a34c02d602\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " Nov 01 22:41:23.923708 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.922543 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17c17145-3afc-4f03-a7a9-6a344420af10-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17c17145-3afc-4f03-a7a9-6a344420af10-openshift-service-ca\") pod \"17c17145-3afc-4f03-a7a9-6a344420af10\" (UID: \"17c17145-3afc-4f03-a7a9-6a344420af10\") " Nov 01 22:41:23.923708 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.922913 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-credential-socket\") pod \"20b6cb03-c7e7-478e-ba20-02a34c02d602\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " Nov 01 22:41:23.923708 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.922974 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-istio-data\") pod \"20b6cb03-c7e7-478e-ba20-02a34c02d602\" (UID: \"20b6cb03-c7e7-478e-ba20-02a34c02d602\") " Nov 01 22:41:23.923708 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.923523 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-workload-socket" pod "20b6cb03-c7e7-478e-ba20-02a34c02d602" (UID: "20b6cb03-c7e7-478e-ba20-02a34c02d602"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:41:23.926727 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.924640 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20b6cb03-c7e7-478e-ba20-02a34c02d602-istiod-ca-cert" pod "20b6cb03-c7e7-478e-ba20-02a34c02d602" (UID: "20b6cb03-c7e7-478e-ba20-02a34c02d602"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:41:23.927377 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.927346 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b6cb03-c7e7-478e-ba20-02a34c02d602-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:23.929215 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.927385 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:23.932154 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.932130 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c17145-3afc-4f03-a7a9-6a344420af10-kube-api-access-tptlf" pod "17c17145-3afc-4f03-a7a9-6a344420af10" (UID: "17c17145-3afc-4f03-a7a9-6a344420af10"). InnerVolumeSpecName "kube-api-access-tptlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:41:23.934323 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.934290 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-istio-envoy" pod "20b6cb03-c7e7-478e-ba20-02a34c02d602" (UID: "20b6cb03-c7e7-478e-ba20-02a34c02d602"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:41:23.937820 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.937505 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20b6cb03-c7e7-478e-ba20-02a34c02d602-istio-podinfo" pod "20b6cb03-c7e7-478e-ba20-02a34c02d602" (UID: "20b6cb03-c7e7-478e-ba20-02a34c02d602"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:41:23.938162 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.938138 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-istio-data" pod "20b6cb03-c7e7-478e-ba20-02a34c02d602" (UID: "20b6cb03-c7e7-478e-ba20-02a34c02d602"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:41:23.938570 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.938539 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-workload-certs" pod "20b6cb03-c7e7-478e-ba20-02a34c02d602" (UID: "20b6cb03-c7e7-478e-ba20-02a34c02d602"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:41:23.940449 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.940383 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17c17145-3afc-4f03-a7a9-6a344420af10-openshift-service-ca" pod "17c17145-3afc-4f03-a7a9-6a344420af10" (UID: "17c17145-3afc-4f03-a7a9-6a344420af10"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:41:23.940547 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.940508 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-credential-socket" pod "20b6cb03-c7e7-478e-ba20-02a34c02d602" (UID: "20b6cb03-c7e7-478e-ba20-02a34c02d602"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:41:23.942467 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.942423 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b6cb03-c7e7-478e-ba20-02a34c02d602-kube-api-access-pqwc4" pod "20b6cb03-c7e7-478e-ba20-02a34c02d602" (UID: "20b6cb03-c7e7-478e-ba20-02a34c02d602"). InnerVolumeSpecName "kube-api-access-pqwc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:41:23.942611 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.942576 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17c17145-3afc-4f03-a7a9-6a344420af10-maas-api-tls" pod "17c17145-3afc-4f03-a7a9-6a344420af10" (UID: "17c17145-3afc-4f03-a7a9-6a344420af10"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:41:23.945896 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.944373 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b6cb03-c7e7-478e-ba20-02a34c02d602-istio-token" pod "20b6cb03-c7e7-478e-ba20-02a34c02d602" (UID: "20b6cb03-c7e7-478e-ba20-02a34c02d602"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:41:23.954614 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:23.951256 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17c17145-3afc-4f03-a7a9-6a344420af10-metrics-certs" pod "17c17145-3afc-4f03-a7a9-6a344420af10" (UID: "17c17145-3afc-4f03-a7a9-6a344420af10"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:41:24.042758 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:24.032326 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tptlf\" (UniqueName: \"kubernetes.io/projected/17c17145-3afc-4f03-a7a9-6a344420af10-kube-api-access-tptlf\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:24.042758 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:24.032361 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqwc4\" (UniqueName: \"kubernetes.io/projected/20b6cb03-c7e7-478e-ba20-02a34c02d602-kube-api-access-pqwc4\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:24.042758 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:24.032386 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b6cb03-c7e7-478e-ba20-02a34c02d602-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:24.042758 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:24.032404 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17c17145-3afc-4f03-a7a9-6a344420af10-metrics-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:24.042758 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:24.032418 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:24.042758 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:24.032432 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b6cb03-c7e7-478e-ba20-02a34c02d602-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:24.042758 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:24.032453 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17c17145-3afc-4f03-a7a9-6a344420af10-maas-api-tls\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:24.042758 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:24.032469 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:24.042758 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:24.032485 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17c17145-3afc-4f03-a7a9-6a344420af10-openshift-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:24.042758 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:24.032498 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:24.042758 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:24.032516 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b6cb03-c7e7-478e-ba20-02a34c02d602-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:24.140165 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:24.140139 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e61a2d4e-7c6d5f9d55-59nj8" Nov 01 22:41:24.236122 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:24.236089 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88140aed-a686-45ba-b920-465c9aef3b53-kube-api-access-855jr\" (UniqueName: \"kubernetes.io/projected/88140aed-a686-45ba-b920-465c9aef3b53-kube-api-access-855jr\") pod \"88140aed-a686-45ba-b920-465c9aef3b53\" (UID: \"88140aed-a686-45ba-b920-465c9aef3b53\") " Nov 01 22:41:24.236122 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:24.236125 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88140aed-a686-45ba-b920-465c9aef3b53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88140aed-a686-45ba-b920-465c9aef3b53-plugins-config-volume\") pod \"88140aed-a686-45ba-b920-465c9aef3b53\" (UID: \"88140aed-a686-45ba-b920-465c9aef3b53\") " Nov 01 22:41:24.236637 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:24.236584 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88140aed-a686-45ba-b920-465c9aef3b53-plugins-config-volume" pod "88140aed-a686-45ba-b920-465c9aef3b53" (UID: "88140aed-a686-45ba-b920-465c9aef3b53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:41:24.237087 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:24.237065 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88140aed-a686-45ba-b920-465c9aef3b53-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:24.239265 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:24.239242 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88140aed-a686-45ba-b920-465c9aef3b53-kube-api-access-855jr" pod "88140aed-a686-45ba-b920-465c9aef3b53" (UID: "88140aed-a686-45ba-b920-465c9aef3b53"). InnerVolumeSpecName "kube-api-access-855jr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:41:24.349945 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:24.345245 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-855jr\" (UniqueName: \"kubernetes.io/projected/88140aed-a686-45ba-b920-465c9aef3b53-kube-api-access-855jr\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:25.087293 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:25.087256 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" event={"ID":"20b6cb03-c7e7-478e-ba20-02a34c02d602","Type":"ContainerDied","Data":"df372c3e29071299577151e93f5d381eb3068f4c033c674648af8c47442783ce"} Nov 01 22:41:25.087865 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:25.087824 5011 scope.go:122] "RemoveContainer" containerID="81f9c40f2e3851ba5d6dfad1138b53d08cb95fcae97e665ee1d2fc6901b58797" Nov 01 22:41:25.088190 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:25.088170 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c" Nov 01 22:41:25.171956 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:25.166508 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e61a2d4e-7c6d5f9d55-59nj8" event={"ID":"88140aed-a686-45ba-b920-465c9aef3b53","Type":"ContainerDied","Data":"12bf2c56e3fdef6350e9d3669e0c9c302d0558164af6621121088e2a3ce5632f"} Nov 01 22:41:25.171956 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:25.166662 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e61a2d4e-7c6d5f9d55-59nj8" Nov 01 22:41:25.283337 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:25.283304 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47" event={"ID":"17c17145-3afc-4f03-a7a9-6a344420af10","Type":"ContainerDied","Data":"cd1dfe6c59773001071f5418234eba3959cba6cf7e5bb94abcbb979104d75594"} Nov 01 22:41:25.283638 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:25.283620 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47" Nov 01 22:41:25.358575 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:25.358364 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e61a2d4e-7c6d5f9d55-59nj8"] Nov 01 22:41:25.372567 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:25.372497 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e61a2d4e-7c6d5f9d55-59nj8"] Nov 01 22:41:25.375675 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:25.375614 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c"] Nov 01 22:41:25.388505 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:25.388455 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-88896ec5-openshift-default-575d9977df-xbm4c"] Nov 01 22:41:25.430528 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:25.430500 5011 scope.go:122] "RemoveContainer" containerID="a639e9cab64c7ee3f3b79493bd1ac74334ab79d5a0e2af60abdf017bb1a06ad9" Nov 01 22:41:25.465469 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:25.465409 5011 scope.go:122] "RemoveContainer" containerID="34d9926e54d855f994f2676ecd7499ccb44fe16f4b3371a8d80b09666a9d68b1" Nov 01 22:41:25.467701 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:25.467678 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47"] Nov 01 22:41:25.471654 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:25.471625 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47"] Nov 01 22:41:25.749485 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:25.749457 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17c17145-3afc-4f03-a7a9-6a344420af10" path="/var/lib/kubelet/pods/17c17145-3afc-4f03-a7a9-6a344420af10/volumes" Nov 01 22:41:25.750055 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:25.750039 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20b6cb03-c7e7-478e-ba20-02a34c02d602" path="/var/lib/kubelet/pods/20b6cb03-c7e7-478e-ba20-02a34c02d602/volumes" Nov 01 22:41:25.750465 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:25.750449 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88140aed-a686-45ba-b920-465c9aef3b53" path="/var/lib/kubelet/pods/88140aed-a686-45ba-b920-465c9aef3b53/volumes" Nov 01 22:41:26.697857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:26.696937 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0f6226e-5dd5496f8b-v6d47" podUID="17c17145-3afc-4f03-a7a9-6a344420af10" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.93:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:30.369846 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:30.369797 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b07f40e8-76cdc77979-2xchf"] Nov 01 22:41:30.370507 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:30.370492 5011 cpu_manager.go:414] "containerMap: removing container" podUID="88140aed-a686-45ba-b920-465c9aef3b53" containerName="payload-pre-processing" Nov 01 22:41:30.370555 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:30.370509 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88140aed-a686-45ba-b920-465c9aef3b53" containerName="payload-pre-processing" Nov 01 22:41:30.370555 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:30.370528 5011 cpu_manager.go:414] "containerMap: removing container" podUID="84e79b24-6ac7-474d-bb27-ec1ce218afbb" containerName="payload-processing" Nov 01 22:41:30.370555 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:30.370532 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84e79b24-6ac7-474d-bb27-ec1ce218afbb" containerName="payload-processing" Nov 01 22:41:30.370555 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:30.370554 5011 cpu_manager.go:414] "containerMap: removing container" podUID="baa15442-fd2c-4bdb-90c8-ff3c043ed73e" containerName="payload-pre-processing" Nov 01 22:41:30.370705 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:30.370560 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baa15442-fd2c-4bdb-90c8-ff3c043ed73e" containerName="payload-pre-processing" Nov 01 22:41:30.370705 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:30.370568 5011 cpu_manager.go:414] "containerMap: removing container" podUID="17c17145-3afc-4f03-a7a9-6a344420af10" containerName="maas-api" Nov 01 22:41:30.370705 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:30.370573 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17c17145-3afc-4f03-a7a9-6a344420af10" containerName="maas-api" Nov 01 22:41:30.370705 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:30.370596 5011 cpu_manager.go:414] "containerMap: removing container" podUID="f0215a01-c33e-4921-b292-88ad9d3dbb31" containerName="payload-pre-processing" Nov 01 22:41:30.370705 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:30.370602 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0215a01-c33e-4921-b292-88ad9d3dbb31" containerName="payload-pre-processing" Nov 01 22:41:30.370705 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:30.370612 5011 cpu_manager.go:414] "containerMap: removing container" podUID="20b6cb03-c7e7-478e-ba20-02a34c02d602" containerName="istio-proxy" Nov 01 22:41:30.370705 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:30.370616 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20b6cb03-c7e7-478e-ba20-02a34c02d602" containerName="istio-proxy" Nov 01 22:41:30.370942 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:30.370719 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="84e79b24-6ac7-474d-bb27-ec1ce218afbb" containerName="payload-processing" Nov 01 22:41:30.370942 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:30.370733 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="baa15442-fd2c-4bdb-90c8-ff3c043ed73e" containerName="payload-pre-processing" Nov 01 22:41:30.370942 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:30.370740 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="17c17145-3afc-4f03-a7a9-6a344420af10" containerName="maas-api" Nov 01 22:41:30.370942 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:30.370748 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0215a01-c33e-4921-b292-88ad9d3dbb31" containerName="payload-pre-processing" Nov 01 22:41:30.370942 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:30.370754 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="20b6cb03-c7e7-478e-ba20-02a34c02d602" containerName="istio-proxy" Nov 01 22:41:30.370942 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:30.370761 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="88140aed-a686-45ba-b920-465c9aef3b53" containerName="payload-pre-processing" Nov 01 22:41:30.376440 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:30.376413 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b07f40e8-76cdc77979-2xchf" Nov 01 22:41:30.399105 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:30.399078 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b07f40e8-76cdc77979-2xchf"] Nov 01 22:41:30.450578 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:30.450532 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b07f40e8\"" Nov 01 22:41:30.450798 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:30.450736 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b07f40e8-dockercfg-mxd9f\"" Nov 01 22:41:30.543862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:30.537756 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br77g\" (UniqueName: \"kubernetes.io/projected/f1542c03-2d27-431e-a260-fc8677eaab07-kube-api-access-br77g\") pod \"payload-processing-e2e-ait-b07f40e8-76cdc77979-2xchf\" (UID: \"f1542c03-2d27-431e-a260-fc8677eaab07\") " pod="openshift-ingress/payload-processing-e2e-ait-b07f40e8-76cdc77979-2xchf" Nov 01 22:41:30.543862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:30.537822 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1542c03-2d27-431e-a260-fc8677eaab07-plugins-config-volume\") pod \"payload-processing-e2e-ait-b07f40e8-76cdc77979-2xchf\" (UID: \"f1542c03-2d27-431e-a260-fc8677eaab07\") " pod="openshift-ingress/payload-processing-e2e-ait-b07f40e8-76cdc77979-2xchf" Nov 01 22:41:30.660938 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:30.657071 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br77g\" (UniqueName: \"kubernetes.io/projected/f1542c03-2d27-431e-a260-fc8677eaab07-kube-api-access-br77g\") pod \"payload-processing-e2e-ait-b07f40e8-76cdc77979-2xchf\" (UID: \"f1542c03-2d27-431e-a260-fc8677eaab07\") " pod="openshift-ingress/payload-processing-e2e-ait-b07f40e8-76cdc77979-2xchf" Nov 01 22:41:30.660938 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:30.657136 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1542c03-2d27-431e-a260-fc8677eaab07-plugins-config-volume\") pod \"payload-processing-e2e-ait-b07f40e8-76cdc77979-2xchf\" (UID: \"f1542c03-2d27-431e-a260-fc8677eaab07\") " pod="openshift-ingress/payload-processing-e2e-ait-b07f40e8-76cdc77979-2xchf" Nov 01 22:41:30.672618 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:30.672586 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1542c03-2d27-431e-a260-fc8677eaab07-plugins-config-volume\") pod \"payload-processing-e2e-ait-b07f40e8-76cdc77979-2xchf\" (UID: \"f1542c03-2d27-431e-a260-fc8677eaab07\") " pod="openshift-ingress/payload-processing-e2e-ait-b07f40e8-76cdc77979-2xchf" Nov 01 22:41:30.726963 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:30.726933 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br77g\" (UniqueName: \"kubernetes.io/projected/f1542c03-2d27-431e-a260-fc8677eaab07-kube-api-access-br77g\") pod \"payload-processing-e2e-ait-b07f40e8-76cdc77979-2xchf\" (UID: \"f1542c03-2d27-431e-a260-fc8677eaab07\") " pod="openshift-ingress/payload-processing-e2e-ait-b07f40e8-76cdc77979-2xchf" Nov 01 22:41:30.992415 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:30.992386 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b07f40e8-76cdc77979-2xchf" Nov 01 22:41:31.284845 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:31.284802 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b07f40e8-76cdc77979-2xchf"] Nov 01 22:41:31.285801 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:41:31.285779 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1542c03_2d27_431e_a260_fc8677eaab07.slice/crio-e7b4f9e3db377acfa0f2e3584f511c61cdad1fd07cd74d075a01937dda4614ba WatchSource:0}: Error finding container e7b4f9e3db377acfa0f2e3584f511c61cdad1fd07cd74d075a01937dda4614ba: Status 404 returned error can't find the container with id e7b4f9e3db377acfa0f2e3584f511c61cdad1fd07cd74d075a01937dda4614ba Nov 01 22:41:31.622407 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:31.622303 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b07f40e8-76cdc77979-2xchf" event={"ID":"f1542c03-2d27-431e-a260-fc8677eaab07","Type":"ContainerStarted","Data":"9f8cdc811a3e8d8af32040dfe3bcaca00a4ea4efaa852c850794df5081181fff"} Nov 01 22:41:31.622407 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:31.622371 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b07f40e8-76cdc77979-2xchf" event={"ID":"f1542c03-2d27-431e-a260-fc8677eaab07","Type":"ContainerStarted","Data":"e7b4f9e3db377acfa0f2e3584f511c61cdad1fd07cd74d075a01937dda4614ba"} Nov 01 22:41:31.622922 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:31.622436 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b07f40e8-76cdc77979-2xchf" Nov 01 22:41:31.673855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:31.660742 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b07f40e8-76cdc77979-2xchf" podStartSLOduration=1.66072111 podStartE2EDuration="1.66072111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:31.657254676 +0000 UTC m=+1060.061845308" watchObservedRunningTime="2026-11-01 22:41:31.66072111 +0000 UTC m=+1060.065311743" Nov 01 22:41:42.632432 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:41:42.632396 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b07f40e8-76cdc77979-2xchf" Nov 01 22:41:46.961305 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:41:46.961260 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:42:05.248030 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:05.245004 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b07f40e8-76cdc77979-2xchf"] Nov 01 22:42:10.246017 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:10.245952 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b07f40e8-76cdc77979-2xchf" podUID="f1542c03-2d27-431e-a260-fc8677eaab07" containerName="payload-processing" containerID="cri-o://9f8cdc811a3e8d8af32040dfe3bcaca00a4ea4efaa852c850794df5081181fff" gracePeriod=25 Nov 01 22:42:10.585858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:10.585736 5011 generic.go:356] "Generic (PLEG): container finished" podID="f1542c03-2d27-431e-a260-fc8677eaab07" containerID="9f8cdc811a3e8d8af32040dfe3bcaca00a4ea4efaa852c850794df5081181fff" exitCode=0 Nov 01 22:42:10.592879 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:10.586304 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b07f40e8-76cdc77979-2xchf" event={"ID":"f1542c03-2d27-431e-a260-fc8677eaab07","Type":"ContainerDied","Data":"9f8cdc811a3e8d8af32040dfe3bcaca00a4ea4efaa852c850794df5081181fff"} Nov 01 22:42:10.680716 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:10.680683 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:42:10.715412 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:10.715373 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b07f40e8-76cdc77979-2xchf" Nov 01 22:42:10.844372 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:10.819172 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1542c03-2d27-431e-a260-fc8677eaab07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1542c03-2d27-431e-a260-fc8677eaab07-plugins-config-volume\") pod \"f1542c03-2d27-431e-a260-fc8677eaab07\" (UID: \"f1542c03-2d27-431e-a260-fc8677eaab07\") " Nov 01 22:42:10.844372 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:10.819482 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1542c03-2d27-431e-a260-fc8677eaab07-kube-api-access-br77g\" (UniqueName: \"kubernetes.io/projected/f1542c03-2d27-431e-a260-fc8677eaab07-kube-api-access-br77g\") pod \"f1542c03-2d27-431e-a260-fc8677eaab07\" (UID: \"f1542c03-2d27-431e-a260-fc8677eaab07\") " Nov 01 22:42:10.844372 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:10.843073 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1542c03-2d27-431e-a260-fc8677eaab07-plugins-config-volume" pod "f1542c03-2d27-431e-a260-fc8677eaab07" (UID: "f1542c03-2d27-431e-a260-fc8677eaab07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:42:10.940864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:10.940108 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1542c03-2d27-431e-a260-fc8677eaab07-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:10.964864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:10.962470 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1542c03-2d27-431e-a260-fc8677eaab07-kube-api-access-br77g" pod "f1542c03-2d27-431e-a260-fc8677eaab07" (UID: "f1542c03-2d27-431e-a260-fc8677eaab07"). InnerVolumeSpecName "kube-api-access-br77g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:42:11.046900 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:11.043791 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br77g\" (UniqueName: \"kubernetes.io/projected/f1542c03-2d27-431e-a260-fc8677eaab07-kube-api-access-br77g\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:11.596537 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:11.596510 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b07f40e8-76cdc77979-2xchf" Nov 01 22:42:11.597033 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:11.596508 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b07f40e8-76cdc77979-2xchf" event={"ID":"f1542c03-2d27-431e-a260-fc8677eaab07","Type":"ContainerDied","Data":"e7b4f9e3db377acfa0f2e3584f511c61cdad1fd07cd74d075a01937dda4614ba"} Nov 01 22:42:11.597033 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:11.596671 5011 scope.go:122] "RemoveContainer" containerID="9f8cdc811a3e8d8af32040dfe3bcaca00a4ea4efaa852c850794df5081181fff" Nov 01 22:42:11.631897 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:11.631870 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b07f40e8-76cdc77979-2xchf"] Nov 01 22:42:11.637023 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:11.636998 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b07f40e8-76cdc77979-2xchf"] Nov 01 22:42:11.753043 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:11.753008 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1542c03-2d27-431e-a260-fc8677eaab07" path="/var/lib/kubelet/pods/f1542c03-2d27-431e-a260-fc8677eaab07/volumes" Nov 01 22:42:16.974167 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:42:16.974125 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:42:35.379959 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:35.379923 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-075771-66579df747-cbnsf"] Nov 01 22:42:35.381064 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:35.381041 5011 cpu_manager.go:414] "containerMap: removing container" podUID="f1542c03-2d27-431e-a260-fc8677eaab07" containerName="payload-processing" Nov 01 22:42:35.381136 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:35.381067 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1542c03-2d27-431e-a260-fc8677eaab07" containerName="payload-processing" Nov 01 22:42:35.381229 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:35.381215 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1542c03-2d27-431e-a260-fc8677eaab07" containerName="payload-processing" Nov 01 22:42:35.386219 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:35.386194 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-075771-66579df747-cbnsf" Nov 01 22:42:35.405817 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:35.405774 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-075771\"" Nov 01 22:42:35.406613 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:35.406582 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-075771-66579df747-cbnsf"] Nov 01 22:42:35.438749 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:35.438691 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqfvw\" (UniqueName: \"kubernetes.io/projected/ec03babc-374f-4704-b397-4db9884f5c77-kube-api-access-vqfvw\") pod \"payload-processing-e2e-shared-b-w5-075771-66579df747-cbnsf\" (UID: \"ec03babc-374f-4704-b397-4db9884f5c77\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-075771-66579df747-cbnsf" Nov 01 22:42:35.439037 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:35.439021 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec03babc-374f-4704-b397-4db9884f5c77-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-075771-66579df747-cbnsf\" (UID: \"ec03babc-374f-4704-b397-4db9884f5c77\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-075771-66579df747-cbnsf" Nov 01 22:42:35.481593 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:35.481548 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-075771-66b777b8dc-cphpc"] Nov 01 22:42:35.486498 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:35.486468 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-075771-66b777b8dc-cphpc" Nov 01 22:42:35.497048 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:35.497017 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-075771-dockercfg-hvs87\"" Nov 01 22:42:35.500893 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:35.500812 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-075771-66b777b8dc-cphpc"] Nov 01 22:42:35.703878 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:35.682707 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec03babc-374f-4704-b397-4db9884f5c77-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-075771-66579df747-cbnsf\" (UID: \"ec03babc-374f-4704-b397-4db9884f5c77\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-075771-66579df747-cbnsf" Nov 01 22:42:35.769854 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:35.674306 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec03babc-374f-4704-b397-4db9884f5c77-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-075771-66579df747-cbnsf\" (UID: \"ec03babc-374f-4704-b397-4db9884f5c77\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-075771-66579df747-cbnsf" Nov 01 22:42:35.769854 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:35.754071 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-075771-66b777b8dc-cphpc\" (UID: \"23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-075771-66b777b8dc-cphpc" Nov 01 22:42:35.769854 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:35.754182 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqfvw\" (UniqueName: \"kubernetes.io/projected/ec03babc-374f-4704-b397-4db9884f5c77-kube-api-access-vqfvw\") pod \"payload-processing-e2e-shared-b-w5-075771-66579df747-cbnsf\" (UID: \"ec03babc-374f-4704-b397-4db9884f5c77\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-075771-66579df747-cbnsf" Nov 01 22:42:35.769854 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:35.754288 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmhvd\" (UniqueName: \"kubernetes.io/projected/23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a-kube-api-access-qmhvd\") pod \"payload-pre-processing-e2e-shared-b-w5-075771-66b777b8dc-cphpc\" (UID: \"23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-075771-66b777b8dc-cphpc" Nov 01 22:42:35.795800 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:35.795768 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqfvw\" (UniqueName: \"kubernetes.io/projected/ec03babc-374f-4704-b397-4db9884f5c77-kube-api-access-vqfvw\") pod \"payload-processing-e2e-shared-b-w5-075771-66579df747-cbnsf\" (UID: \"ec03babc-374f-4704-b397-4db9884f5c77\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-075771-66579df747-cbnsf" Nov 01 22:42:35.862857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:35.857279 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmhvd\" (UniqueName: \"kubernetes.io/projected/23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a-kube-api-access-qmhvd\") pod \"payload-pre-processing-e2e-shared-b-w5-075771-66b777b8dc-cphpc\" (UID: \"23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-075771-66b777b8dc-cphpc" Nov 01 22:42:35.864447 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:35.864413 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-075771-66b777b8dc-cphpc\" (UID: \"23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-075771-66b777b8dc-cphpc" Nov 01 22:42:35.872916 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:35.865784 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-075771-66b777b8dc-cphpc\" (UID: \"23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-075771-66b777b8dc-cphpc" Nov 01 22:42:35.906863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:35.906667 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmhvd\" (UniqueName: \"kubernetes.io/projected/23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a-kube-api-access-qmhvd\") pod \"payload-pre-processing-e2e-shared-b-w5-075771-66b777b8dc-cphpc\" (UID: \"23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-075771-66b777b8dc-cphpc" Nov 01 22:42:36.004478 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:36.004120 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-075771-66579df747-cbnsf" Nov 01 22:42:36.103192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:36.103130 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-075771-66b777b8dc-cphpc" Nov 01 22:42:36.403998 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:36.402781 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-075771-66579df747-cbnsf"] Nov 01 22:42:36.411777 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:42:36.411721 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec03babc_374f_4704_b397_4db9884f5c77.slice/crio-3a4fb97e8bba12a654ca28739399b2aeb7881946fb791e7de6ecd13f008c01ca WatchSource:0}: Error finding container 3a4fb97e8bba12a654ca28739399b2aeb7881946fb791e7de6ecd13f008c01ca: Status 404 returned error can't find the container with id 3a4fb97e8bba12a654ca28739399b2aeb7881946fb791e7de6ecd13f008c01ca Nov 01 22:42:36.449818 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:36.449750 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-075771-66b777b8dc-cphpc"] Nov 01 22:42:36.452275 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:42:36.451653 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d1d2d2_11d8_4ca4_b08a_7ba3713b3b1a.slice/crio-1520326e83f9a8a9127ce82790063ef64e6b05da3682d8254ae022e9f83f1fec WatchSource:0}: Error finding container 1520326e83f9a8a9127ce82790063ef64e6b05da3682d8254ae022e9f83f1fec: Status 404 returned error can't find the container with id 1520326e83f9a8a9127ce82790063ef64e6b05da3682d8254ae022e9f83f1fec Nov 01 22:42:37.059381 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:37.048672 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-075771-66579df747-cbnsf" event={"ID":"ec03babc-374f-4704-b397-4db9884f5c77","Type":"ContainerStarted","Data":"0c2b7cb9bf75dadb0e28ae351067ed1ebddb0dedb1334d410d52bbd9f97934b6"} Nov 01 22:42:37.059381 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:37.048791 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-075771-66579df747-cbnsf" event={"ID":"ec03babc-374f-4704-b397-4db9884f5c77","Type":"ContainerStarted","Data":"3a4fb97e8bba12a654ca28739399b2aeb7881946fb791e7de6ecd13f008c01ca"} Nov 01 22:42:37.059381 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:37.050144 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-075771-66579df747-cbnsf" Nov 01 22:42:37.083853 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:37.081794 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-075771-66b777b8dc-cphpc" event={"ID":"23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a","Type":"ContainerStarted","Data":"a5950948ce19aeb9563b993d3c68e955013658294a2dde375aacb6c1644157f4"} Nov 01 22:42:37.083853 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:37.081888 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-075771-66b777b8dc-cphpc" event={"ID":"23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a","Type":"ContainerStarted","Data":"1520326e83f9a8a9127ce82790063ef64e6b05da3682d8254ae022e9f83f1fec"} Nov 01 22:42:37.083853 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:37.083247 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-075771-66b777b8dc-cphpc" Nov 01 22:42:37.107856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:37.106329 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-075771-66579df747-cbnsf" podStartSLOduration=2.106291587 podStartE2EDuration="2.106291587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:37.080172837 +0000 UTC m=+1125.484763471" watchObservedRunningTime="2026-11-01 22:42:37.106291587 +0000 UTC m=+1125.510882220" Nov 01 22:42:37.181909 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:37.181414 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-075771-66b777b8dc-cphpc" podStartSLOduration=2.181389359 podStartE2EDuration="2.181389359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:37.115204223 +0000 UTC m=+1125.519794856" watchObservedRunningTime="2026-11-01 22:42:37.181389359 +0000 UTC m=+1125.585980032" Nov 01 22:42:43.447483 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:43.447449 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb"] Nov 01 22:42:43.452736 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:43.452713 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb" Nov 01 22:42:43.471586 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:43.471560 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-075771-serving-cert\"" Nov 01 22:42:43.479472 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:43.479443 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb"] Nov 01 22:42:43.586892 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:43.586860 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f45f8128-5315-4bd0-98ed-7f0b57309295-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb\" (UID: \"f45f8128-5315-4bd0-98ed-7f0b57309295\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb" Nov 01 22:42:43.587096 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:43.587015 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f45f8128-5315-4bd0-98ed-7f0b57309295-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb\" (UID: \"f45f8128-5315-4bd0-98ed-7f0b57309295\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb" Nov 01 22:42:43.587162 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:43.587141 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f45f8128-5315-4bd0-98ed-7f0b57309295-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb\" (UID: \"f45f8128-5315-4bd0-98ed-7f0b57309295\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb" Nov 01 22:42:43.587308 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:43.587291 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jpv5\" (UniqueName: \"kubernetes.io/projected/f45f8128-5315-4bd0-98ed-7f0b57309295-kube-api-access-5jpv5\") pod \"maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb\" (UID: \"f45f8128-5315-4bd0-98ed-7f0b57309295\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb" Nov 01 22:42:43.697904 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:43.694338 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f45f8128-5315-4bd0-98ed-7f0b57309295-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb\" (UID: \"f45f8128-5315-4bd0-98ed-7f0b57309295\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb" Nov 01 22:42:43.697904 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:43.694469 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f45f8128-5315-4bd0-98ed-7f0b57309295-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb\" (UID: \"f45f8128-5315-4bd0-98ed-7f0b57309295\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb" Nov 01 22:42:43.697904 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:43.694571 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f45f8128-5315-4bd0-98ed-7f0b57309295-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb\" (UID: \"f45f8128-5315-4bd0-98ed-7f0b57309295\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb" Nov 01 22:42:43.697904 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:43.694766 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jpv5\" (UniqueName: \"kubernetes.io/projected/f45f8128-5315-4bd0-98ed-7f0b57309295-kube-api-access-5jpv5\") pod \"maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb\" (UID: \"f45f8128-5315-4bd0-98ed-7f0b57309295\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb" Nov 01 22:42:43.697904 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:43.696199 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f45f8128-5315-4bd0-98ed-7f0b57309295-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb\" (UID: \"f45f8128-5315-4bd0-98ed-7f0b57309295\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb" Nov 01 22:42:43.702701 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:43.699616 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f45f8128-5315-4bd0-98ed-7f0b57309295-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb\" (UID: \"f45f8128-5315-4bd0-98ed-7f0b57309295\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb" Nov 01 22:42:43.707574 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:43.707549 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f45f8128-5315-4bd0-98ed-7f0b57309295-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb\" (UID: \"f45f8128-5315-4bd0-98ed-7f0b57309295\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb" Nov 01 22:42:43.742073 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:43.736281 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jpv5\" (UniqueName: \"kubernetes.io/projected/f45f8128-5315-4bd0-98ed-7f0b57309295-kube-api-access-5jpv5\") pod \"maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb\" (UID: \"f45f8128-5315-4bd0-98ed-7f0b57309295\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb" Nov 01 22:42:43.787856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:43.782341 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb" Nov 01 22:42:44.267862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:44.266758 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb"] Nov 01 22:42:45.275057 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:45.274962 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb" event={"ID":"f45f8128-5315-4bd0-98ed-7f0b57309295","Type":"ContainerStarted","Data":"689fa041d4286a4209a45cdcf7db2c59d87f58dc47d279177e7865df0c4fe695"} Nov 01 22:42:45.275057 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:45.275051 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb" event={"ID":"f45f8128-5315-4bd0-98ed-7f0b57309295","Type":"ContainerStarted","Data":"336e297f7a841e3f94bb07b0ad66b1a2de2be799213410d689a1a4fadc0bc7e4"} Nov 01 22:42:45.275476 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:45.275131 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb" Nov 01 22:42:45.297226 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:45.297175 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb" podStartSLOduration=1.645284821 podStartE2EDuration="2.297161231s" totalImagesPullingTime="651.87641ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:45.293666784 +0000 UTC m=+1133.698257441" watchObservedRunningTime="2026-11-01 22:42:45.297161231 +0000 UTC m=+1133.701751918" Nov 01 22:42:47.088147 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:42:47.088102 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:42:47.633703 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:47.633661 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb"] Nov 01 22:42:47.645972 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:47.645928 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb" Nov 01 22:42:47.650933 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:47.649548 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb"] Nov 01 22:42:47.655981 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:47.655366 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-dc635e-serving-cert\"" Nov 01 22:42:47.740907 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:47.737721 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae1a941e-5ef1-46b4-8948-70b23ff5b300-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb\" (UID: \"ae1a941e-5ef1-46b4-8948-70b23ff5b300\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb" Nov 01 22:42:47.740907 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:47.737801 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae1a941e-5ef1-46b4-8948-70b23ff5b300-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb\" (UID: \"ae1a941e-5ef1-46b4-8948-70b23ff5b300\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb" Nov 01 22:42:47.740907 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:47.738134 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w55rf\" (UniqueName: \"kubernetes.io/projected/ae1a941e-5ef1-46b4-8948-70b23ff5b300-kube-api-access-w55rf\") pod \"maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb\" (UID: \"ae1a941e-5ef1-46b4-8948-70b23ff5b300\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb" Nov 01 22:42:47.740907 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:47.738182 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae1a941e-5ef1-46b4-8948-70b23ff5b300-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb\" (UID: \"ae1a941e-5ef1-46b4-8948-70b23ff5b300\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb" Nov 01 22:42:47.853873 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:47.839582 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae1a941e-5ef1-46b4-8948-70b23ff5b300-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb\" (UID: \"ae1a941e-5ef1-46b4-8948-70b23ff5b300\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb" Nov 01 22:42:47.853873 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:47.839995 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w55rf\" (UniqueName: \"kubernetes.io/projected/ae1a941e-5ef1-46b4-8948-70b23ff5b300-kube-api-access-w55rf\") pod \"maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb\" (UID: \"ae1a941e-5ef1-46b4-8948-70b23ff5b300\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb" Nov 01 22:42:47.853873 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:47.840046 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae1a941e-5ef1-46b4-8948-70b23ff5b300-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb\" (UID: \"ae1a941e-5ef1-46b4-8948-70b23ff5b300\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb" Nov 01 22:42:47.853873 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:47.840135 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae1a941e-5ef1-46b4-8948-70b23ff5b300-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb\" (UID: \"ae1a941e-5ef1-46b4-8948-70b23ff5b300\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb" Nov 01 22:42:47.853873 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:42:47.840338 5011 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dc635e-serving-cert: secret "maas-api-e2e-shared-b-w0-dc635e-serving-cert" not found Nov 01 22:42:47.853873 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:42:47.840433 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae1a941e-5ef1-46b4-8948-70b23ff5b300-maas-api-tls podName:ae1a941e-5ef1-46b4-8948-70b23ff5b300 nodeName:}" failed. No retries permitted until 2026-11-01 22:42:48.340408926 +0000 UTC m=+1136.744999555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ae1a941e-5ef1-46b4-8948-70b23ff5b300-maas-api-tls") pod "maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb" (UID: "ae1a941e-5ef1-46b4-8948-70b23ff5b300") : secret "maas-api-e2e-shared-b-w0-dc635e-serving-cert" not found Nov 01 22:42:47.853873 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:47.842467 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae1a941e-5ef1-46b4-8948-70b23ff5b300-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb\" (UID: \"ae1a941e-5ef1-46b4-8948-70b23ff5b300\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb" Nov 01 22:42:47.862863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:47.860791 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae1a941e-5ef1-46b4-8948-70b23ff5b300-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb\" (UID: \"ae1a941e-5ef1-46b4-8948-70b23ff5b300\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb" Nov 01 22:42:47.869998 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:47.869930 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w55rf\" (UniqueName: \"kubernetes.io/projected/ae1a941e-5ef1-46b4-8948-70b23ff5b300-kube-api-access-w55rf\") pod \"maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb\" (UID: \"ae1a941e-5ef1-46b4-8948-70b23ff5b300\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb" Nov 01 22:42:48.352867 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:48.350566 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae1a941e-5ef1-46b4-8948-70b23ff5b300-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb\" (UID: \"ae1a941e-5ef1-46b4-8948-70b23ff5b300\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb" Nov 01 22:42:48.396856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:48.385692 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae1a941e-5ef1-46b4-8948-70b23ff5b300-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb\" (UID: \"ae1a941e-5ef1-46b4-8948-70b23ff5b300\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb" Nov 01 22:42:48.571301 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:48.571270 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb" Nov 01 22:42:48.881816 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:48.881776 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb"] Nov 01 22:42:49.100571 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:49.100545 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-075771-66b777b8dc-cphpc" Nov 01 22:42:49.100730 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:49.100618 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-075771-66579df747-cbnsf" Nov 01 22:42:49.356521 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:49.356493 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb" event={"ID":"ae1a941e-5ef1-46b4-8948-70b23ff5b300","Type":"ContainerStarted","Data":"3301762b154df0c216dd60d60e01ec952fa4c93d5b42b45d8e4b73c30d340a7f"} Nov 01 22:42:50.365931 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:50.365891 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb" event={"ID":"ae1a941e-5ef1-46b4-8948-70b23ff5b300","Type":"ContainerStarted","Data":"7f3b3a97f0e009d95a013d186fc5db1bed6ebe0e8a6d9ce4a887403010ef8a57"} Nov 01 22:42:50.366356 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:50.365986 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb" Nov 01 22:42:50.388716 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:50.388662 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb" podStartSLOduration=2.96158167 podStartE2EDuration="3.388646617s" totalImagesPullingTime="427.064947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:50.385507615 +0000 UTC m=+1138.790098250" watchObservedRunningTime="2026-11-01 22:42:50.388646617 +0000 UTC m=+1138.793237248" Nov 01 22:42:51.288694 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:51.288667 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb" Nov 01 22:42:56.429093 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:42:56.429056 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb" Nov 01 22:43:06.268191 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:06.268151 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:43:11.065859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:11.064956 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7jkf4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:11.065859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:11.065060 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" podUID="5ffbc1a2-7268-4ab3-a81c-e04b42fc441f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:11.119858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:11.118955 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jkvvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:11.119858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:11.119035 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" podUID="1b6bec10-9ac2-4708-9631-b76c801c0c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:16.961122 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:43:16.961080 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:43:34.569079 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:34.564196 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4534ad-666786f5b7-t2kpt"] Nov 01 22:43:34.588506 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:34.586759 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4534ad-666786f5b7-t2kpt" Nov 01 22:43:34.594888 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:34.592420 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-4534ad\"" Nov 01 22:43:34.614427 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:34.586658 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4534ad-666786f5b7-t2kpt"] Nov 01 22:43:34.666857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:34.642225 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-4534ad-dockercfg-l56mq\"" Nov 01 22:43:34.689520 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:34.677950 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv2q4\" (UniqueName: \"kubernetes.io/projected/22dc49f2-d19a-4ff8-957f-e333317212a8-kube-api-access-cv2q4\") pod \"payload-pre-processing-e2e-api-a-4534ad-666786f5b7-t2kpt\" (UID: \"22dc49f2-d19a-4ff8-957f-e333317212a8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4534ad-666786f5b7-t2kpt" Nov 01 22:43:34.698862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:34.690164 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22dc49f2-d19a-4ff8-957f-e333317212a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4534ad-666786f5b7-t2kpt\" (UID: \"22dc49f2-d19a-4ff8-957f-e333317212a8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4534ad-666786f5b7-t2kpt" Nov 01 22:43:34.881796 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:34.826602 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22dc49f2-d19a-4ff8-957f-e333317212a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4534ad-666786f5b7-t2kpt\" (UID: \"22dc49f2-d19a-4ff8-957f-e333317212a8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4534ad-666786f5b7-t2kpt" Nov 01 22:43:34.881796 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:34.826806 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv2q4\" (UniqueName: \"kubernetes.io/projected/22dc49f2-d19a-4ff8-957f-e333317212a8-kube-api-access-cv2q4\") pod \"payload-pre-processing-e2e-api-a-4534ad-666786f5b7-t2kpt\" (UID: \"22dc49f2-d19a-4ff8-957f-e333317212a8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4534ad-666786f5b7-t2kpt" Nov 01 22:43:34.913952 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:34.882991 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22dc49f2-d19a-4ff8-957f-e333317212a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4534ad-666786f5b7-t2kpt\" (UID: \"22dc49f2-d19a-4ff8-957f-e333317212a8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4534ad-666786f5b7-t2kpt" Nov 01 22:43:34.948255 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:34.948219 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv2q4\" (UniqueName: \"kubernetes.io/projected/22dc49f2-d19a-4ff8-957f-e333317212a8-kube-api-access-cv2q4\") pod \"payload-pre-processing-e2e-api-a-4534ad-666786f5b7-t2kpt\" (UID: \"22dc49f2-d19a-4ff8-957f-e333317212a8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4534ad-666786f5b7-t2kpt" Nov 01 22:43:34.959532 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:34.959497 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4534ad-666786f5b7-t2kpt" Nov 01 22:43:35.992206 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:35.992159 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4534ad-666786f5b7-t2kpt"] Nov 01 22:43:36.176944 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:36.175820 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4534ad-666786f5b7-t2kpt" event={"ID":"22dc49f2-d19a-4ff8-957f-e333317212a8","Type":"ContainerStarted","Data":"d8f3cc2c5f48313fd5fd26fbd0f7ddc86585890ee504decea1d58277f5ac215e"} Nov 01 22:43:36.176944 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:36.175920 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4534ad-666786f5b7-t2kpt" event={"ID":"22dc49f2-d19a-4ff8-957f-e333317212a8","Type":"ContainerStarted","Data":"06574b0824ad25ff3ddde4e9bbf73880799911435c53e84012d58e8508a2d45d"} Nov 01 22:43:36.176944 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:36.176902 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4534ad-666786f5b7-t2kpt" Nov 01 22:43:36.198210 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:36.198143 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4534ad-666786f5b7-t2kpt" podStartSLOduration=2.198124687 podStartE2EDuration="2.198124687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:36.19437499 +0000 UTC m=+1184.598965625" watchObservedRunningTime="2026-11-01 22:43:36.198124687 +0000 UTC m=+1184.602715319" Nov 01 22:43:45.317608 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:45.317571 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7"] Nov 01 22:43:45.322104 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:45.322084 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7" Nov 01 22:43:45.331802 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:45.331780 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7"] Nov 01 22:43:45.368633 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:45.368588 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-4534ad-serving-cert\"" Nov 01 22:43:45.373131 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:45.373093 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rxpm\" (UniqueName: \"kubernetes.io/projected/7f568e19-ff42-4413-a06f-34f9f9594077-kube-api-access-2rxpm\") pod \"maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7\" (UID: \"7f568e19-ff42-4413-a06f-34f9f9594077\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7" Nov 01 22:43:45.373339 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:45.373312 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f568e19-ff42-4413-a06f-34f9f9594077-openshift-service-ca\") pod \"maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7\" (UID: \"7f568e19-ff42-4413-a06f-34f9f9594077\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7" Nov 01 22:43:45.373598 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:45.373576 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f568e19-ff42-4413-a06f-34f9f9594077-maas-api-tls\") pod \"maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7\" (UID: \"7f568e19-ff42-4413-a06f-34f9f9594077\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7" Nov 01 22:43:45.373676 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:45.373614 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f568e19-ff42-4413-a06f-34f9f9594077-metrics-certs\") pod \"maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7\" (UID: \"7f568e19-ff42-4413-a06f-34f9f9594077\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7" Nov 01 22:43:45.474611 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:45.474568 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f568e19-ff42-4413-a06f-34f9f9594077-openshift-service-ca\") pod \"maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7\" (UID: \"7f568e19-ff42-4413-a06f-34f9f9594077\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7" Nov 01 22:43:45.474786 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:45.474746 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f568e19-ff42-4413-a06f-34f9f9594077-maas-api-tls\") pod \"maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7\" (UID: \"7f568e19-ff42-4413-a06f-34f9f9594077\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7" Nov 01 22:43:45.474786 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:45.474768 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f568e19-ff42-4413-a06f-34f9f9594077-metrics-certs\") pod \"maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7\" (UID: \"7f568e19-ff42-4413-a06f-34f9f9594077\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7" Nov 01 22:43:45.474908 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:45.474894 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rxpm\" (UniqueName: \"kubernetes.io/projected/7f568e19-ff42-4413-a06f-34f9f9594077-kube-api-access-2rxpm\") pod \"maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7\" (UID: \"7f568e19-ff42-4413-a06f-34f9f9594077\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7" Nov 01 22:43:45.475914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:45.475882 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f568e19-ff42-4413-a06f-34f9f9594077-openshift-service-ca\") pod \"maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7\" (UID: \"7f568e19-ff42-4413-a06f-34f9f9594077\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7" Nov 01 22:43:45.477522 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:45.477498 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f568e19-ff42-4413-a06f-34f9f9594077-metrics-certs\") pod \"maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7\" (UID: \"7f568e19-ff42-4413-a06f-34f9f9594077\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7" Nov 01 22:43:45.477634 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:45.477520 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f568e19-ff42-4413-a06f-34f9f9594077-maas-api-tls\") pod \"maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7\" (UID: \"7f568e19-ff42-4413-a06f-34f9f9594077\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7" Nov 01 22:43:45.484330 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:45.484307 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rxpm\" (UniqueName: \"kubernetes.io/projected/7f568e19-ff42-4413-a06f-34f9f9594077-kube-api-access-2rxpm\") pod \"maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7\" (UID: \"7f568e19-ff42-4413-a06f-34f9f9594077\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7" Nov 01 22:43:45.640910 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:45.638917 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7" Nov 01 22:43:46.183855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:46.183715 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7"] Nov 01 22:43:46.362278 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:46.362190 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7" event={"ID":"7f568e19-ff42-4413-a06f-34f9f9594077","Type":"ContainerStarted","Data":"fbe61b5593bf1827be5f25f191165d24f101722ee96bcd9f40015846b7d2c84a"} Nov 01 22:43:46.967109 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:43:46.967067 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:43:47.422928 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:47.422885 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7" event={"ID":"7f568e19-ff42-4413-a06f-34f9f9594077","Type":"ContainerStarted","Data":"10f168e05d6bf075acbd6e9b1cf26438447da9bd38521908059267a1dd066e44"} Nov 01 22:43:47.423426 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:47.423023 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7" Nov 01 22:43:47.442613 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:47.442543 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7" podStartSLOduration=1.90316766 podStartE2EDuration="2.442527648s" totalImagesPullingTime="539.359988ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:47.441469747 +0000 UTC m=+1195.846060380" watchObservedRunningTime="2026-11-01 22:43:47.442527648 +0000 UTC m=+1195.847118280" Nov 01 22:43:48.259190 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:48.259159 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4534ad-666786f5b7-t2kpt" Nov 01 22:43:53.447133 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:43:53.447102 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7" Nov 01 22:44:04.305196 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:04.305157 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-df4b43-886976559-l6xmq"] Nov 01 22:44:04.318581 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:04.318545 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-df4b43-886976559-l6xmq"] Nov 01 22:44:04.318742 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:04.318659 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-df4b43-886976559-l6xmq" Nov 01 22:44:04.337266 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:04.337238 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87414a3d-0620-434b-9736-32dd6ef0a8e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-df4b43-886976559-l6xmq\" (UID: \"87414a3d-0620-434b-9736-32dd6ef0a8e1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-df4b43-886976559-l6xmq" Nov 01 22:44:04.337584 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:04.337528 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq6wm\" (UniqueName: \"kubernetes.io/projected/87414a3d-0620-434b-9736-32dd6ef0a8e1-kube-api-access-zq6wm\") pod \"payload-pre-processing-e2e-api-b-df4b43-886976559-l6xmq\" (UID: \"87414a3d-0620-434b-9736-32dd6ef0a8e1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-df4b43-886976559-l6xmq" Nov 01 22:44:04.426968 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:04.425316 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-df4b43\"" Nov 01 22:44:04.426968 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:04.425514 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-df4b43-dockercfg-v2g2j\"" Nov 01 22:44:04.439335 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:04.439293 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87414a3d-0620-434b-9736-32dd6ef0a8e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-df4b43-886976559-l6xmq\" (UID: \"87414a3d-0620-434b-9736-32dd6ef0a8e1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-df4b43-886976559-l6xmq" Nov 01 22:44:04.452553 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:04.452515 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq6wm\" (UniqueName: \"kubernetes.io/projected/87414a3d-0620-434b-9736-32dd6ef0a8e1-kube-api-access-zq6wm\") pod \"payload-pre-processing-e2e-api-b-df4b43-886976559-l6xmq\" (UID: \"87414a3d-0620-434b-9736-32dd6ef0a8e1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-df4b43-886976559-l6xmq" Nov 01 22:44:04.453646 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:04.453619 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87414a3d-0620-434b-9736-32dd6ef0a8e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-df4b43-886976559-l6xmq\" (UID: \"87414a3d-0620-434b-9736-32dd6ef0a8e1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-df4b43-886976559-l6xmq" Nov 01 22:44:04.504854 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:04.500437 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq6wm\" (UniqueName: \"kubernetes.io/projected/87414a3d-0620-434b-9736-32dd6ef0a8e1-kube-api-access-zq6wm\") pod \"payload-pre-processing-e2e-api-b-df4b43-886976559-l6xmq\" (UID: \"87414a3d-0620-434b-9736-32dd6ef0a8e1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-df4b43-886976559-l6xmq" Nov 01 22:44:04.638261 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:04.636648 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-df4b43-886976559-l6xmq" Nov 01 22:44:05.094319 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:05.094275 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-df4b43-886976559-l6xmq"] Nov 01 22:44:05.094676 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:44:05.094653 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87414a3d_0620_434b_9736_32dd6ef0a8e1.slice/crio-2d6945e35d2303767e18528ab48c8beacd35f6b6204d46b0ef6b6cc66537986e WatchSource:0}: Error finding container 2d6945e35d2303767e18528ab48c8beacd35f6b6204d46b0ef6b6cc66537986e: Status 404 returned error can't find the container with id 2d6945e35d2303767e18528ab48c8beacd35f6b6204d46b0ef6b6cc66537986e Nov 01 22:44:05.749969 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:05.749933 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-df4b43-886976559-l6xmq" event={"ID":"87414a3d-0620-434b-9736-32dd6ef0a8e1","Type":"ContainerStarted","Data":"24e02e783cfb5921a0d96e647dfa8659afefb74b55b33ba01c52e22db59d2082"} Nov 01 22:44:05.749969 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:05.749994 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-df4b43-886976559-l6xmq" event={"ID":"87414a3d-0620-434b-9736-32dd6ef0a8e1","Type":"ContainerStarted","Data":"2d6945e35d2303767e18528ab48c8beacd35f6b6204d46b0ef6b6cc66537986e"} Nov 01 22:44:05.750492 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:05.750026 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-df4b43-886976559-l6xmq" Nov 01 22:44:05.782327 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:05.782260 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-df4b43-886976559-l6xmq" podStartSLOduration=1.782098722 podStartE2EDuration="1.782098722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:05.772357053 +0000 UTC m=+1214.176947677" watchObservedRunningTime="2026-11-01 22:44:05.782098722 +0000 UTC m=+1214.186689395" Nov 01 22:44:08.108842 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.108610 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r"] Nov 01 22:44:08.124931 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.124897 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r"] Nov 01 22:44:08.125116 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.125039 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:08.136412 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.136349 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2a5c3234-openshift-default-dockercfg-kj6td\"" Nov 01 22:44:08.300865 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.294964 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-credential-socket\") pod \"e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:08.300865 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.295021 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-workload-certs\") pod \"e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:08.300865 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.295065 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istio-data\") pod \"e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:08.308876 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.307110 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istiod-ca-cert\") pod \"e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:08.308876 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.307272 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg8sl\" (UniqueName: \"kubernetes.io/projected/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-kube-api-access-bg8sl\") pod \"e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:08.308876 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.307374 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istio-token\") pod \"e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:08.308876 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.307532 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istio-envoy\") pod \"e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:08.308876 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.307582 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-workload-socket\") pod \"e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:08.308876 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.307681 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istio-podinfo\") pod \"e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:08.413905 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.410176 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istio-podinfo\") pod \"e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:08.413905 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.410285 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-credential-socket\") pod \"e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:08.413905 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.410311 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-workload-certs\") pod \"e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:08.413905 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.410351 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istio-data\") pod \"e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:08.413905 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.410433 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istiod-ca-cert\") pod \"e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:08.413905 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.410545 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg8sl\" (UniqueName: \"kubernetes.io/projected/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-kube-api-access-bg8sl\") pod \"e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:08.413905 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.410647 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istio-token\") pod \"e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:08.413905 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.410792 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istio-envoy\") pod \"e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:08.413905 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.410857 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-workload-socket\") pod \"e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:08.413905 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.411368 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-workload-socket\") pod \"e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:08.413905 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.412489 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istiod-ca-cert\") pod \"e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:08.413905 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.412722 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-credential-socket\") pod \"e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:08.425941 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.420902 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istio-podinfo\") pod \"e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:08.425941 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.423613 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istio-data\") pod \"e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:08.432963 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.432935 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istio-envoy\") pod \"e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:08.433164 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.433063 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-workload-certs\") pod \"e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:08.439869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.436582 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istio-token\") pod \"e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:08.446756 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.446714 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg8sl\" (UniqueName: \"kubernetes.io/projected/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-kube-api-access-bg8sl\") pod \"e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:08.451238 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.451214 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:08.980970 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.980934 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r"] Nov 01 22:44:08.981777 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:44:08.981749 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58c1620e_7736_4fe3_bfde_8024cfb8d2e5.slice/crio-c71a95b2fd179057e493b20b5f8472fb5c4bd6d7ab6731682aff357864444829 WatchSource:0}: Error finding container c71a95b2fd179057e493b20b5f8472fb5c4bd6d7ab6731682aff357864444829: Status 404 returned error can't find the container with id c71a95b2fd179057e493b20b5f8472fb5c4bd6d7ab6731682aff357864444829 Nov 01 22:44:08.985136 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.985099 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:44:08.985220 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.985173 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:44:08.985220 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:08.985202 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:44:09.790104 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:09.790066 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" event={"ID":"58c1620e-7736-4fe3-bfde-8024cfb8d2e5","Type":"ContainerStarted","Data":"dcce2c0ba8383a16bd9638aa080632c45f034cb3bc7780fbb9fcbd47213854f5"} Nov 01 22:44:09.790676 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:09.790130 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" event={"ID":"58c1620e-7736-4fe3-bfde-8024cfb8d2e5","Type":"ContainerStarted","Data":"c71a95b2fd179057e493b20b5f8472fb5c4bd6d7ab6731682aff357864444829"} Nov 01 22:44:09.827762 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:09.827703 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" podStartSLOduration=1.827689026 podStartE2EDuration="1.827689026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:09.827520204 +0000 UTC m=+1218.232110833" watchObservedRunningTime="2026-11-01 22:44:09.827689026 +0000 UTC m=+1218.232279658" Nov 01 22:44:10.452511 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:10.452480 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:10.458980 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:10.458952 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:10.826919 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:10.801147 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:10.826919 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:10.825163 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:44:11.251888 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:11.249634 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a9f94e-57fdf95d6c-szfm7"] Nov 01 22:44:11.255589 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:11.255544 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a9f94e-57fdf95d6c-szfm7" Nov 01 22:44:11.260706 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:11.260676 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a9f94e-57fdf95d6c-szfm7"] Nov 01 22:44:11.448008 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:11.425179 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-a9f94e\"" Nov 01 22:44:11.448008 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:11.425321 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-a9f94e-dockercfg-h8hz6\"" Nov 01 22:44:11.448008 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:11.429012 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0beecaf1-fcc2-4d74-ba44-005ecf40b355-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-a9f94e-57fdf95d6c-szfm7\" (UID: \"0beecaf1-fcc2-4d74-ba44-005ecf40b355\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a9f94e-57fdf95d6c-szfm7" Nov 01 22:44:11.448008 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:11.429081 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vhbq\" (UniqueName: \"kubernetes.io/projected/0beecaf1-fcc2-4d74-ba44-005ecf40b355-kube-api-access-2vhbq\") pod \"payload-processing-e2e-shared-a-w3-a9f94e-57fdf95d6c-szfm7\" (UID: \"0beecaf1-fcc2-4d74-ba44-005ecf40b355\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a9f94e-57fdf95d6c-szfm7" Nov 01 22:44:11.535917 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:11.532692 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0beecaf1-fcc2-4d74-ba44-005ecf40b355-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-a9f94e-57fdf95d6c-szfm7\" (UID: \"0beecaf1-fcc2-4d74-ba44-005ecf40b355\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a9f94e-57fdf95d6c-szfm7" Nov 01 22:44:11.535917 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:11.532763 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vhbq\" (UniqueName: \"kubernetes.io/projected/0beecaf1-fcc2-4d74-ba44-005ecf40b355-kube-api-access-2vhbq\") pod \"payload-processing-e2e-shared-a-w3-a9f94e-57fdf95d6c-szfm7\" (UID: \"0beecaf1-fcc2-4d74-ba44-005ecf40b355\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a9f94e-57fdf95d6c-szfm7" Nov 01 22:44:11.587181 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:11.587142 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0beecaf1-fcc2-4d74-ba44-005ecf40b355-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-a9f94e-57fdf95d6c-szfm7\" (UID: \"0beecaf1-fcc2-4d74-ba44-005ecf40b355\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a9f94e-57fdf95d6c-szfm7" Nov 01 22:44:11.675293 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:11.668946 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vhbq\" (UniqueName: \"kubernetes.io/projected/0beecaf1-fcc2-4d74-ba44-005ecf40b355-kube-api-access-2vhbq\") pod \"payload-processing-e2e-shared-a-w3-a9f94e-57fdf95d6c-szfm7\" (UID: \"0beecaf1-fcc2-4d74-ba44-005ecf40b355\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a9f94e-57fdf95d6c-szfm7" Nov 01 22:44:11.876035 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:11.875941 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a9f94e-57fdf95d6c-szfm7" Nov 01 22:44:12.164254 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:12.164223 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a9f94e-57fdf95d6c-szfm7"] Nov 01 22:44:12.164722 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:44:12.164696 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0beecaf1_fcc2_4d74_ba44_005ecf40b355.slice/crio-28a272dc4b19a88c33220b30659d53a6ff6da236e4fad55e0ec4c45c0bff3d50 WatchSource:0}: Error finding container 28a272dc4b19a88c33220b30659d53a6ff6da236e4fad55e0ec4c45c0bff3d50: Status 404 returned error can't find the container with id 28a272dc4b19a88c33220b30659d53a6ff6da236e4fad55e0ec4c45c0bff3d50 Nov 01 22:44:12.857603 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:12.853230 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a9f94e-57fdf95d6c-szfm7" event={"ID":"0beecaf1-fcc2-4d74-ba44-005ecf40b355","Type":"ContainerStarted","Data":"b8300abb8b9986e6ec85bb5138fc9bce64590037e39ce93373a6d2622ce66d05"} Nov 01 22:44:12.857603 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:12.853319 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a9f94e-57fdf95d6c-szfm7" event={"ID":"0beecaf1-fcc2-4d74-ba44-005ecf40b355","Type":"ContainerStarted","Data":"28a272dc4b19a88c33220b30659d53a6ff6da236e4fad55e0ec4c45c0bff3d50"} Nov 01 22:44:12.857603 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:12.853486 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a9f94e-57fdf95d6c-szfm7" Nov 01 22:44:13.685893 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:13.685795 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a9f94e-57fdf95d6c-szfm7" podStartSLOduration=2.685778496 podStartE2EDuration="2.685778496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:12.890856696 +0000 UTC m=+1221.295447343" watchObservedRunningTime="2026-11-01 22:44:13.685778496 +0000 UTC m=+1222.090369129" Nov 01 22:44:13.707856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:13.707288 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:44:14.304992 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:14.304954 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2a5c3234-7c548f45c7-x6thk"] Nov 01 22:44:14.315966 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:14.315925 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2a5c3234-7c548f45c7-x6thk" Nov 01 22:44:14.329882 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:14.327763 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2a5c3234-7c548f45c7-x6thk"] Nov 01 22:44:14.334090 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:14.334068 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2a5c3234\"" Nov 01 22:44:14.334244 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:14.334224 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2a5c3234-dockercfg-jfkcl\"" Nov 01 22:44:14.465855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:14.426427 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53c8acf3-27ab-4ed5-87aa-1d64e8e015ab-plugins-config-volume\") pod \"payload-processing-e2e-mt-2a5c3234-7c548f45c7-x6thk\" (UID: \"53c8acf3-27ab-4ed5-87aa-1d64e8e015ab\") " pod="openshift-ingress/payload-processing-e2e-mt-2a5c3234-7c548f45c7-x6thk" Nov 01 22:44:14.465855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:14.426507 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8mw8\" (UniqueName: \"kubernetes.io/projected/53c8acf3-27ab-4ed5-87aa-1d64e8e015ab-kube-api-access-r8mw8\") pod \"payload-processing-e2e-mt-2a5c3234-7c548f45c7-x6thk\" (UID: \"53c8acf3-27ab-4ed5-87aa-1d64e8e015ab\") " pod="openshift-ingress/payload-processing-e2e-mt-2a5c3234-7c548f45c7-x6thk" Nov 01 22:44:14.764914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:14.699330 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53c8acf3-27ab-4ed5-87aa-1d64e8e015ab-plugins-config-volume\") pod \"payload-processing-e2e-mt-2a5c3234-7c548f45c7-x6thk\" (UID: \"53c8acf3-27ab-4ed5-87aa-1d64e8e015ab\") " pod="openshift-ingress/payload-processing-e2e-mt-2a5c3234-7c548f45c7-x6thk" Nov 01 22:44:14.764914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:14.699421 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8mw8\" (UniqueName: \"kubernetes.io/projected/53c8acf3-27ab-4ed5-87aa-1d64e8e015ab-kube-api-access-r8mw8\") pod \"payload-processing-e2e-mt-2a5c3234-7c548f45c7-x6thk\" (UID: \"53c8acf3-27ab-4ed5-87aa-1d64e8e015ab\") " pod="openshift-ingress/payload-processing-e2e-mt-2a5c3234-7c548f45c7-x6thk" Nov 01 22:44:14.876883 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:14.876817 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53c8acf3-27ab-4ed5-87aa-1d64e8e015ab-plugins-config-volume\") pod \"payload-processing-e2e-mt-2a5c3234-7c548f45c7-x6thk\" (UID: \"53c8acf3-27ab-4ed5-87aa-1d64e8e015ab\") " pod="openshift-ingress/payload-processing-e2e-mt-2a5c3234-7c548f45c7-x6thk" Nov 01 22:44:14.908055 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:14.893782 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8mw8\" (UniqueName: \"kubernetes.io/projected/53c8acf3-27ab-4ed5-87aa-1d64e8e015ab-kube-api-access-r8mw8\") pod \"payload-processing-e2e-mt-2a5c3234-7c548f45c7-x6thk\" (UID: \"53c8acf3-27ab-4ed5-87aa-1d64e8e015ab\") " pod="openshift-ingress/payload-processing-e2e-mt-2a5c3234-7c548f45c7-x6thk" Nov 01 22:44:14.960885 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:14.943432 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2a5c3234-7c548f45c7-x6thk" Nov 01 22:44:16.117910 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:16.098075 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2a5c3234-7c548f45c7-x6thk"] Nov 01 22:44:16.757229 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:16.757199 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-df4b43-886976559-l6xmq" Nov 01 22:44:16.970677 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:44:16.970642 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:44:17.104419 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:17.104318 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2a5c3234-7c548f45c7-x6thk" event={"ID":"53c8acf3-27ab-4ed5-87aa-1d64e8e015ab","Type":"ContainerStarted","Data":"9bc5caf5778fe8281d263a28dda86c8a623cc5d2bf81ff6af5cf466d4dcaf2f7"} Nov 01 22:44:17.104419 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:17.104382 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2a5c3234-7c548f45c7-x6thk" event={"ID":"53c8acf3-27ab-4ed5-87aa-1d64e8e015ab","Type":"ContainerStarted","Data":"210ca37160937aa312ca994fdb4b8e295c0eca79a0e4256c8b5d2f48d48b31d8"} Nov 01 22:44:17.106753 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:17.106729 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-2a5c3234-7c548f45c7-x6thk" Nov 01 22:44:17.132705 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:17.132643 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-2a5c3234-7c548f45c7-x6thk" podStartSLOduration=3.132625705 podStartE2EDuration="3.132625705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:17.129208214 +0000 UTC m=+1225.533798848" watchObservedRunningTime="2026-11-01 22:44:17.132625705 +0000 UTC m=+1225.537216338" Nov 01 22:44:25.066539 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:25.066511 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a9f94e-57fdf95d6c-szfm7" Nov 01 22:44:26.636711 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:26.636670 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c"] Nov 01 22:44:26.641996 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:26.641971 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c" Nov 01 22:44:26.653006 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:26.652978 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-a9f94e-serving-cert\"" Nov 01 22:44:26.660282 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:26.660255 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c"] Nov 01 22:44:26.905035 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:26.904869 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c\" (UID: \"9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c" Nov 01 22:44:26.936413 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:26.936370 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c\" (UID: \"9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c" Nov 01 22:44:26.965854 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:26.963249 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c\" (UID: \"9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c" Nov 01 22:44:26.965854 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:26.964034 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkrx4\" (UniqueName: \"kubernetes.io/projected/9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9-kube-api-access-hkrx4\") pod \"maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c\" (UID: \"9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c" Nov 01 22:44:27.065683 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:27.065644 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c\" (UID: \"9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c" Nov 01 22:44:27.065683 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:27.065689 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c\" (UID: \"9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c" Nov 01 22:44:27.066026 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:27.065770 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkrx4\" (UniqueName: \"kubernetes.io/projected/9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9-kube-api-access-hkrx4\") pod \"maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c\" (UID: \"9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c" Nov 01 22:44:27.066026 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:27.065800 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c\" (UID: \"9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c" Nov 01 22:44:27.066467 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:27.066439 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c\" (UID: \"9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c" Nov 01 22:44:27.068581 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:27.068554 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c\" (UID: \"9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c" Nov 01 22:44:27.068674 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:27.068591 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c\" (UID: \"9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c" Nov 01 22:44:27.077414 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:27.077391 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkrx4\" (UniqueName: \"kubernetes.io/projected/9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9-kube-api-access-hkrx4\") pod \"maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c\" (UID: \"9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c" Nov 01 22:44:27.290855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:27.287560 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c" Nov 01 22:44:27.968981 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:27.968950 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c"] Nov 01 22:44:27.969624 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:44:27.969599 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c0fb1a7_6aab_4fb7_9352_ed1d6272b1e9.slice/crio-aa6cd3667a4498f3fd371107be0965f8fd83606f5f34621f3c841f96dba759fa WatchSource:0}: Error finding container aa6cd3667a4498f3fd371107be0965f8fd83606f5f34621f3c841f96dba759fa: Status 404 returned error can't find the container with id aa6cd3667a4498f3fd371107be0965f8fd83606f5f34621f3c841f96dba759fa Nov 01 22:44:28.521260 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:28.521223 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c" event={"ID":"9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9","Type":"ContainerStarted","Data":"c4d79389d01b11e7409cac776c78eb0da8ce192a5782f9d2368f0586e78cdfe4"} Nov 01 22:44:28.521464 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:28.521292 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c" event={"ID":"9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9","Type":"ContainerStarted","Data":"aa6cd3667a4498f3fd371107be0965f8fd83606f5f34621f3c841f96dba759fa"} Nov 01 22:44:28.521464 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:28.521429 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c" Nov 01 22:44:28.543215 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:28.543165 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c" podStartSLOduration=2.148188821 podStartE2EDuration="2.543150402s" totalImagesPullingTime="394.961581ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:28.54048381 +0000 UTC m=+1236.945074446" watchObservedRunningTime="2026-11-01 22:44:28.543150402 +0000 UTC m=+1236.947741033" Nov 01 22:44:29.281630 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:29.281603 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-2a5c3234-7c548f45c7-x6thk" Nov 01 22:44:34.534203 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:34.534169 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c" Nov 01 22:44:39.516862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:39.510657 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv"] Nov 01 22:44:39.516862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:39.511007 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" podUID="dccd06ae-f4d0-485f-abe7-89cc6e34a431" containerName="istio-proxy" containerID="cri-o://d6228b94f0a9bed0f8a9a0ad888a3634ffa204a3ee67fd44fe038363d574f9ca" gracePeriod=30 Nov 01 22:44:39.537862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:39.520812 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w"] Nov 01 22:44:39.537862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:39.521679 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" podUID="b8318b18-986e-4435-8518-42d01c66b6b4" containerName="istio-proxy" containerID="cri-o://e25506f889102a6e7a09d747d53bf50317ddb28962d1b084f9b1b1103c73a583" gracePeriod=30 Nov 01 22:44:39.547847 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:39.545236 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx"] Nov 01 22:44:39.547847 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:39.545501 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" podUID="0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" containerName="istio-proxy" containerID="cri-o://6d48c5c86ec2056e63654073d273714adca2a4cfdddc093daf449a807b2b2017" gracePeriod=30 Nov 01 22:44:43.774865 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:43.769203 5011 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:44:43.774865 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:43.769265 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" podUID="0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:44:44.953555 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:44.953329 5011 generic.go:356] "Generic (PLEG): container finished" podID="b8318b18-986e-4435-8518-42d01c66b6b4" containerID="e25506f889102a6e7a09d747d53bf50317ddb28962d1b084f9b1b1103c73a583" exitCode=0 Nov 01 22:44:44.953555 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:44.953491 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" event={"ID":"b8318b18-986e-4435-8518-42d01c66b6b4","Type":"ContainerDied","Data":"e25506f889102a6e7a09d747d53bf50317ddb28962d1b084f9b1b1103c73a583"} Nov 01 22:44:44.956053 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:44.955925 5011 generic.go:356] "Generic (PLEG): container finished" podID="0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" containerID="6d48c5c86ec2056e63654073d273714adca2a4cfdddc093daf449a807b2b2017" exitCode=0 Nov 01 22:44:44.956053 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:44.955984 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" event={"ID":"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef","Type":"ContainerDied","Data":"6d48c5c86ec2056e63654073d273714adca2a4cfdddc093daf449a807b2b2017"} Nov 01 22:44:44.958257 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:44.958233 5011 generic.go:356] "Generic (PLEG): container finished" podID="dccd06ae-f4d0-485f-abe7-89cc6e34a431" containerID="d6228b94f0a9bed0f8a9a0ad888a3634ffa204a3ee67fd44fe038363d574f9ca" exitCode=0 Nov 01 22:44:44.958367 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:44.958282 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" event={"ID":"dccd06ae-f4d0-485f-abe7-89cc6e34a431","Type":"ContainerDied","Data":"d6228b94f0a9bed0f8a9a0ad888a3634ffa204a3ee67fd44fe038363d574f9ca"} Nov 01 22:44:45.308041 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.296124 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:44:45.359405 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.356593 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-workload-socket\") pod \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " Nov 01 22:44:45.359405 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.356684 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istio-data\") pod \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " Nov 01 22:44:45.359405 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.356765 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dccd06ae-f4d0-485f-abe7-89cc6e34a431-kube-api-access-xbzmv\" (UniqueName: \"kubernetes.io/projected/dccd06ae-f4d0-485f-abe7-89cc6e34a431-kube-api-access-xbzmv\") pod \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " Nov 01 22:44:45.359405 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.356799 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-credential-socket\") pod \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " Nov 01 22:44:45.359405 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.356863 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istiod-ca-cert\") pod \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " Nov 01 22:44:45.359405 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.356912 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istio-token\" (UniqueName: \"kubernetes.io/projected/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istio-token\") pod \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " Nov 01 22:44:45.359405 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.356966 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istio-envoy\") pod \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " Nov 01 22:44:45.359405 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.357053 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-workload-certs\") pod \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " Nov 01 22:44:45.359405 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.357279 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istio-podinfo\") pod \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\" (UID: \"dccd06ae-f4d0-485f-abe7-89cc6e34a431\") " Nov 01 22:44:45.360028 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.359550 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istio-data" pod "dccd06ae-f4d0-485f-abe7-89cc6e34a431" (UID: "dccd06ae-f4d0-485f-abe7-89cc6e34a431"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:45.361034 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.360402 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-workload-certs" pod "dccd06ae-f4d0-485f-abe7-89cc6e34a431" (UID: "dccd06ae-f4d0-485f-abe7-89cc6e34a431"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:45.361034 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.360472 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-credential-socket" pod "dccd06ae-f4d0-485f-abe7-89cc6e34a431" (UID: "dccd06ae-f4d0-485f-abe7-89cc6e34a431"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:45.361034 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.360789 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-workload-socket" pod "dccd06ae-f4d0-485f-abe7-89cc6e34a431" (UID: "dccd06ae-f4d0-485f-abe7-89cc6e34a431"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:45.361034 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.361002 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istiod-ca-cert" pod "dccd06ae-f4d0-485f-abe7-89cc6e34a431" (UID: "dccd06ae-f4d0-485f-abe7-89cc6e34a431"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:45.364969 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.364937 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:45.365110 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.364971 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:45.365110 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.364985 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:45.365110 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.364998 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:45.365110 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.365011 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:45.369147 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.368586 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dccd06ae-f4d0-485f-abe7-89cc6e34a431-kube-api-access-xbzmv" pod "dccd06ae-f4d0-485f-abe7-89cc6e34a431" (UID: "dccd06ae-f4d0-485f-abe7-89cc6e34a431"). InnerVolumeSpecName "kube-api-access-xbzmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:45.369147 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.369105 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istio-token" pod "dccd06ae-f4d0-485f-abe7-89cc6e34a431" (UID: "dccd06ae-f4d0-485f-abe7-89cc6e34a431"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:45.369414 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.369356 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istio-podinfo" pod "dccd06ae-f4d0-485f-abe7-89cc6e34a431" (UID: "dccd06ae-f4d0-485f-abe7-89cc6e34a431"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:44:45.370983 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.369734 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istio-envoy" pod "dccd06ae-f4d0-485f-abe7-89cc6e34a431" (UID: "dccd06ae-f4d0-485f-abe7-89cc6e34a431"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:45.421534 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.419976 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:44:45.460913 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.460791 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:44:45.466722 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.466330 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-credential-socket\") pod \"b8318b18-986e-4435-8518-42d01c66b6b4\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " Nov 01 22:44:45.466722 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.466374 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8318b18-986e-4435-8518-42d01c66b6b4-kube-api-access-v8n9c\" (UniqueName: \"kubernetes.io/projected/b8318b18-986e-4435-8518-42d01c66b6b4-kube-api-access-v8n9c\") pod \"b8318b18-986e-4435-8518-42d01c66b6b4\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " Nov 01 22:44:45.466722 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.466448 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8318b18-986e-4435-8518-42d01c66b6b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8318b18-986e-4435-8518-42d01c66b6b4-istiod-ca-cert\") pod \"b8318b18-986e-4435-8518-42d01c66b6b4\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " Nov 01 22:44:45.466722 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.466487 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-istio-data\") pod \"b8318b18-986e-4435-8518-42d01c66b6b4\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " Nov 01 22:44:45.467104 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.466797 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8318b18-986e-4435-8518-42d01c66b6b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8318b18-986e-4435-8518-42d01c66b6b4-istio-podinfo\") pod \"b8318b18-986e-4435-8518-42d01c66b6b4\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " Nov 01 22:44:45.467104 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.467044 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-workload-certs\") pod \"b8318b18-986e-4435-8518-42d01c66b6b4\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " Nov 01 22:44:45.467220 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.467142 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-workload-socket\") pod \"b8318b18-986e-4435-8518-42d01c66b6b4\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " Nov 01 22:44:45.467278 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.467231 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-istio-envoy\") pod \"b8318b18-986e-4435-8518-42d01c66b6b4\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " Nov 01 22:44:45.467337 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.467287 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8318b18-986e-4435-8518-42d01c66b6b4-istio-token\" (UniqueName: \"kubernetes.io/projected/b8318b18-986e-4435-8518-42d01c66b6b4-istio-token\") pod \"b8318b18-986e-4435-8518-42d01c66b6b4\" (UID: \"b8318b18-986e-4435-8518-42d01c66b6b4\") " Nov 01 22:44:45.468214 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.468184 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbzmv\" (UniqueName: \"kubernetes.io/projected/dccd06ae-f4d0-485f-abe7-89cc6e34a431-kube-api-access-xbzmv\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:45.468335 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.468218 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:45.468335 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.468231 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:45.468335 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.468244 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dccd06ae-f4d0-485f-abe7-89cc6e34a431-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:45.468504 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.468413 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-istio-data" pod "b8318b18-986e-4435-8518-42d01c66b6b4" (UID: "b8318b18-986e-4435-8518-42d01c66b6b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:45.469682 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.468619 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-credential-socket" pod "b8318b18-986e-4435-8518-42d01c66b6b4" (UID: "b8318b18-986e-4435-8518-42d01c66b6b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:45.469682 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.468812 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-workload-certs" pod "b8318b18-986e-4435-8518-42d01c66b6b4" (UID: "b8318b18-986e-4435-8518-42d01c66b6b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:45.470236 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.470017 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8318b18-986e-4435-8518-42d01c66b6b4-istiod-ca-cert" pod "b8318b18-986e-4435-8518-42d01c66b6b4" (UID: "b8318b18-986e-4435-8518-42d01c66b6b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:45.470682 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.470349 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-workload-socket" pod "b8318b18-986e-4435-8518-42d01c66b6b4" (UID: "b8318b18-986e-4435-8518-42d01c66b6b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:45.471705 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.471642 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8318b18-986e-4435-8518-42d01c66b6b4-kube-api-access-v8n9c" pod "b8318b18-986e-4435-8518-42d01c66b6b4" (UID: "b8318b18-986e-4435-8518-42d01c66b6b4"). InnerVolumeSpecName "kube-api-access-v8n9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:45.471909 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.471885 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8318b18-986e-4435-8518-42d01c66b6b4-istio-podinfo" pod "b8318b18-986e-4435-8518-42d01c66b6b4" (UID: "b8318b18-986e-4435-8518-42d01c66b6b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:44:45.472721 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.472688 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-istio-envoy" pod "b8318b18-986e-4435-8518-42d01c66b6b4" (UID: "b8318b18-986e-4435-8518-42d01c66b6b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:45.473879 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.473824 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8318b18-986e-4435-8518-42d01c66b6b4-istio-token" pod "b8318b18-986e-4435-8518-42d01c66b6b4" (UID: "b8318b18-986e-4435-8518-42d01c66b6b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:45.569552 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.569515 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istio-data\") pod \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " Nov 01 22:44:45.569755 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.569569 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istio-podinfo\") pod \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " Nov 01 22:44:45.569755 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.569617 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-workload-socket\") pod \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " Nov 01 22:44:45.569755 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.569681 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istio-token\" (UniqueName: \"kubernetes.io/projected/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istio-token\") pod \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " Nov 01 22:44:45.569755 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.569718 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-workload-certs\") pod \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " Nov 01 22:44:45.569991 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.569778 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-kube-api-access-8k6k2\" (UniqueName: \"kubernetes.io/projected/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-kube-api-access-8k6k2\") pod \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " Nov 01 22:44:45.569991 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.569898 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istio-data" pod "0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" (UID: "0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:45.569991 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.569981 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-credential-socket\") pod \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " Nov 01 22:44:45.570149 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.569997 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-workload-socket" pod "0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" (UID: "0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:45.570149 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.570070 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istio-envoy\") pod \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " Nov 01 22:44:45.570149 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.570085 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-workload-certs" pod "0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" (UID: "0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:45.570149 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.570123 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istiod-ca-cert\") pod \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\" (UID: \"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef\") " Nov 01 22:44:45.570331 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.570222 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-credential-socket" pod "0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" (UID: "0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:45.570758 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.570725 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istiod-ca-cert" pod "0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" (UID: "0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:45.571275 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.571251 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8318b18-986e-4435-8518-42d01c66b6b4-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:45.571392 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.571282 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:45.571392 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.571296 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:45.571392 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.571310 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:45.571392 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.571323 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:45.571392 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.571337 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:45.571392 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.571350 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:45.571392 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.571363 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8318b18-986e-4435-8518-42d01c66b6b4-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:45.571392 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.571376 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:45.571392 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.571391 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8n9c\" (UniqueName: \"kubernetes.io/projected/b8318b18-986e-4435-8518-42d01c66b6b4-kube-api-access-v8n9c\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:45.571929 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.571406 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8318b18-986e-4435-8518-42d01c66b6b4-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:45.571929 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.571418 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8318b18-986e-4435-8518-42d01c66b6b4-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:45.571929 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.571431 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:45.571929 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.571447 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:45.572282 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.572257 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istio-podinfo" pod "0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" (UID: "0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:44:45.572381 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.572354 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istio-token" pod "0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" (UID: "0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:45.572381 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.572365 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-kube-api-access-8k6k2" pod "0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" (UID: "0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef"). InnerVolumeSpecName "kube-api-access-8k6k2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:45.572719 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.572697 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istio-envoy" pod "0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" (UID: "0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:44:45.673398 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.673369 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:45.673398 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.673397 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:45.673590 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.673409 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:45.673590 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.673423 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k6k2\" (UniqueName: \"kubernetes.io/projected/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef-kube-api-access-8k6k2\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:45.968456 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.968425 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" Nov 01 22:44:45.968972 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.968425 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv" event={"ID":"dccd06ae-f4d0-485f-abe7-89cc6e34a431","Type":"ContainerDied","Data":"820d6e0754c71df0d52b9e881cd5f7f91453d3c741158ca06f58fa76b74fbcbd"} Nov 01 22:44:45.968972 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.968611 5011 scope.go:122] "RemoveContainer" containerID="d6228b94f0a9bed0f8a9a0ad888a3634ffa204a3ee67fd44fe038363d574f9ca" Nov 01 22:44:45.979495 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.979080 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" event={"ID":"b8318b18-986e-4435-8518-42d01c66b6b4","Type":"ContainerDied","Data":"2569002280c0338d85a4b1c96793b56365af27ba103db428645607de439771dd"} Nov 01 22:44:45.981152 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.979207 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w" Nov 01 22:44:45.982486 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.982462 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" event={"ID":"0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef","Type":"ContainerDied","Data":"38b8cd760b55f1ee2456ef389e4de7cc98d3377634ef30cb6bc3c2602583e6d0"} Nov 01 22:44:45.982615 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.982554 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx" Nov 01 22:44:45.998853 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.995169 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt"] Nov 01 22:44:45.998853 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.996451 5011 cpu_manager.go:414] "containerMap: removing container" podUID="b8318b18-986e-4435-8518-42d01c66b6b4" containerName="istio-proxy" Nov 01 22:44:45.998853 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.996471 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8318b18-986e-4435-8518-42d01c66b6b4" containerName="istio-proxy" Nov 01 22:44:45.998853 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.996519 5011 cpu_manager.go:414] "containerMap: removing container" podUID="dccd06ae-f4d0-485f-abe7-89cc6e34a431" containerName="istio-proxy" Nov 01 22:44:45.998853 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:45.996526 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dccd06ae-f4d0-485f-abe7-89cc6e34a431" containerName="istio-proxy" Nov 01 22:44:46.013246 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.013216 5011 cpu_manager.go:414] "containerMap: removing container" podUID="0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" containerName="istio-proxy" Nov 01 22:44:46.013246 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.013245 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" containerName="istio-proxy" Nov 01 22:44:46.013988 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.013793 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="dccd06ae-f4d0-485f-abe7-89cc6e34a431" containerName="istio-proxy" Nov 01 22:44:46.013988 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.013816 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" containerName="istio-proxy" Nov 01 22:44:46.013988 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.013871 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8318b18-986e-4435-8518-42d01c66b6b4" containerName="istio-proxy" Nov 01 22:44:46.045712 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.040940 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt"] Nov 01 22:44:46.045712 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.041024 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:46.045712 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.044930 5011 scope.go:122] "RemoveContainer" containerID="e25506f889102a6e7a09d747d53bf50317ddb28962d1b084f9b1b1103c73a583" Nov 01 22:44:46.049099 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.048782 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-56b833-openshift-default-dockercfg-grknh\"" Nov 01 22:44:46.064878 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.064788 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w"] Nov 01 22:44:46.085857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.074915 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-qgc5w"] Nov 01 22:44:46.113864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.092889 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knv4g\" (UniqueName: \"kubernetes.io/projected/55ecde9f-2ac4-4791-ada0-184f9d6141f8-kube-api-access-knv4g\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:46.113864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.092957 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-credential-socket\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:46.113864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.093112 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-workload-socket\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:46.113864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.093188 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istio-envoy\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:46.113864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.093227 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istio-token\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:46.113864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.093272 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istio-podinfo\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:46.113864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.093345 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istio-data\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:46.113864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.093385 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-workload-certs\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:46.113864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.093423 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istiod-ca-cert\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:46.214855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.211084 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istio-data\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:46.214855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.211145 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-workload-certs\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:46.214855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.211184 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istiod-ca-cert\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:46.214855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.211362 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knv4g\" (UniqueName: \"kubernetes.io/projected/55ecde9f-2ac4-4791-ada0-184f9d6141f8-kube-api-access-knv4g\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:46.214855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.211403 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-credential-socket\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:46.214855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.211540 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-workload-socket\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:46.214855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.211612 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istio-envoy\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:46.214855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.211645 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istio-token\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:46.214855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.211689 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istio-podinfo\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:46.214855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.213378 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istiod-ca-cert\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:46.214855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.213642 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-workload-socket\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:46.214855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.213648 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istio-data\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:46.214855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.213891 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-workload-certs\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:46.351970 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.303213 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-credential-socket\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:46.392111 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.357781 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istio-envoy\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:46.392111 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.390858 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istio-podinfo\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:46.392111 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.391921 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv"] Nov 01 22:44:46.471420 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.420650 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knv4g\" (UniqueName: \"kubernetes.io/projected/55ecde9f-2ac4-4791-ada0-184f9d6141f8-kube-api-access-knv4g\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:46.471420 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.458248 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-225tv"] Nov 01 22:44:46.567495 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.567457 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istio-token\") pod \"e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:46.568794 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.568648 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:46.577934 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.577904 5011 scope.go:122] "RemoveContainer" containerID="6d48c5c86ec2056e63654073d273714adca2a4cfdddc093daf449a807b2b2017" Nov 01 22:44:46.668852 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.665325 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx"] Nov 01 22:44:46.676298 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:46.675181 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-4g9dx"] Nov 01 22:44:47.055879 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:47.055433 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt"] Nov 01 22:44:47.074850 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:47.074794 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:44:47.075024 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:47.074903 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:44:47.075024 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:47.074947 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:44:47.094088 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:44:47.094055 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:44:47.811069 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:47.805575 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef" path="/var/lib/kubelet/pods/0ff9691c-dbd5-41e2-bd2d-b7ae79c2cbef/volumes" Nov 01 22:44:47.830860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:47.824645 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8318b18-986e-4435-8518-42d01c66b6b4" path="/var/lib/kubelet/pods/b8318b18-986e-4435-8518-42d01c66b6b4/volumes" Nov 01 22:44:47.830860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:47.825494 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dccd06ae-f4d0-485f-abe7-89cc6e34a431" path="/var/lib/kubelet/pods/dccd06ae-f4d0-485f-abe7-89cc6e34a431/volumes" Nov 01 22:44:48.084785 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:48.084681 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" event={"ID":"55ecde9f-2ac4-4791-ada0-184f9d6141f8","Type":"ContainerStarted","Data":"61dcae3adeec8abf26fe5998cbd979dde6ea70f158b6ac5014f02b02dbac20a0"} Nov 01 22:44:48.084785 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:48.084739 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" event={"ID":"55ecde9f-2ac4-4791-ada0-184f9d6141f8","Type":"ContainerStarted","Data":"03b4ca3c700538a047e3d2290aa73817a0e46a9cb5da3c2f19e367c9f9ed5244"} Nov 01 22:44:48.109110 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:48.109056 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" podStartSLOduration=3.109041806 podStartE2EDuration="3.109041806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:48.104491182 +0000 UTC m=+1256.509081818" watchObservedRunningTime="2026-11-01 22:44:48.109041806 +0000 UTC m=+1256.513632437" Nov 01 22:44:48.570010 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:48.569965 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:48.639591 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:48.639562 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:49.092238 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:49.092205 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:49.093321 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:49.093299 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 22:44:51.578856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:51.559195 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4534ad-666786f5b7-t2kpt"] Nov 01 22:44:52.303951 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:52.303914 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56b833-556fbfdf9-snb4s"] Nov 01 22:44:52.312119 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:52.312058 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56b833-556fbfdf9-snb4s" Nov 01 22:44:52.319952 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:52.319908 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-56b833\"" Nov 01 22:44:52.321909 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:52.321880 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56b833-556fbfdf9-snb4s"] Nov 01 22:44:52.380721 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:52.380683 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-56b833-d86bfc667-g45bg"] Nov 01 22:44:52.389366 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:52.389334 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56b833-d86bfc667-g45bg" Nov 01 22:44:52.391097 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:52.391057 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-56b833-d86bfc667-g45bg"] Nov 01 22:44:52.396061 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:52.396036 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-56b833-dockercfg-4ftrm\"" Nov 01 22:44:52.453855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:52.452194 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpb4p\" (UniqueName: \"kubernetes.io/projected/a187f76d-d133-49a8-9614-6dcd3fe2f1b0-kube-api-access-wpb4p\") pod \"payload-processing-e2e-shared-b-w3-56b833-d86bfc667-g45bg\" (UID: \"a187f76d-d133-49a8-9614-6dcd3fe2f1b0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56b833-d86bfc667-g45bg" Nov 01 22:44:52.453855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:52.452320 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/212e6260-16ba-4d02-9f5d-84cf58050476-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-56b833-556fbfdf9-snb4s\" (UID: \"212e6260-16ba-4d02-9f5d-84cf58050476\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56b833-556fbfdf9-snb4s" Nov 01 22:44:52.453855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:52.452413 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a187f76d-d133-49a8-9614-6dcd3fe2f1b0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-56b833-d86bfc667-g45bg\" (UID: \"a187f76d-d133-49a8-9614-6dcd3fe2f1b0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56b833-d86bfc667-g45bg" Nov 01 22:44:52.453855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:52.452567 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfph6\" (UniqueName: \"kubernetes.io/projected/212e6260-16ba-4d02-9f5d-84cf58050476-kube-api-access-lfph6\") pod \"payload-pre-processing-e2e-shared-b-w3-56b833-556fbfdf9-snb4s\" (UID: \"212e6260-16ba-4d02-9f5d-84cf58050476\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56b833-556fbfdf9-snb4s" Nov 01 22:44:52.712855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:52.711680 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/212e6260-16ba-4d02-9f5d-84cf58050476-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-56b833-556fbfdf9-snb4s\" (UID: \"212e6260-16ba-4d02-9f5d-84cf58050476\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56b833-556fbfdf9-snb4s" Nov 01 22:44:52.712855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:52.712143 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a187f76d-d133-49a8-9614-6dcd3fe2f1b0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-56b833-d86bfc667-g45bg\" (UID: \"a187f76d-d133-49a8-9614-6dcd3fe2f1b0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56b833-d86bfc667-g45bg" Nov 01 22:44:52.712855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:52.712493 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfph6\" (UniqueName: \"kubernetes.io/projected/212e6260-16ba-4d02-9f5d-84cf58050476-kube-api-access-lfph6\") pod \"payload-pre-processing-e2e-shared-b-w3-56b833-556fbfdf9-snb4s\" (UID: \"212e6260-16ba-4d02-9f5d-84cf58050476\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56b833-556fbfdf9-snb4s" Nov 01 22:44:52.713458 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:52.713077 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpb4p\" (UniqueName: \"kubernetes.io/projected/a187f76d-d133-49a8-9614-6dcd3fe2f1b0-kube-api-access-wpb4p\") pod \"payload-processing-e2e-shared-b-w3-56b833-d86bfc667-g45bg\" (UID: \"a187f76d-d133-49a8-9614-6dcd3fe2f1b0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56b833-d86bfc667-g45bg" Nov 01 22:44:52.747059 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:52.714569 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/212e6260-16ba-4d02-9f5d-84cf58050476-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-56b833-556fbfdf9-snb4s\" (UID: \"212e6260-16ba-4d02-9f5d-84cf58050476\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56b833-556fbfdf9-snb4s" Nov 01 22:44:52.747317 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:52.738555 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a187f76d-d133-49a8-9614-6dcd3fe2f1b0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-56b833-d86bfc667-g45bg\" (UID: \"a187f76d-d133-49a8-9614-6dcd3fe2f1b0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56b833-d86bfc667-g45bg" Nov 01 22:44:52.855890 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:52.855066 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfph6\" (UniqueName: \"kubernetes.io/projected/212e6260-16ba-4d02-9f5d-84cf58050476-kube-api-access-lfph6\") pod \"payload-pre-processing-e2e-shared-b-w3-56b833-556fbfdf9-snb4s\" (UID: \"212e6260-16ba-4d02-9f5d-84cf58050476\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56b833-556fbfdf9-snb4s" Nov 01 22:44:52.902856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:52.902753 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpb4p\" (UniqueName: \"kubernetes.io/projected/a187f76d-d133-49a8-9614-6dcd3fe2f1b0-kube-api-access-wpb4p\") pod \"payload-processing-e2e-shared-b-w3-56b833-d86bfc667-g45bg\" (UID: \"a187f76d-d133-49a8-9614-6dcd3fe2f1b0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56b833-d86bfc667-g45bg" Nov 01 22:44:52.942687 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:52.942207 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56b833-556fbfdf9-snb4s" Nov 01 22:44:53.074517 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:53.072801 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56b833-d86bfc667-g45bg" Nov 01 22:44:53.901502 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:53.901349 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56b833-556fbfdf9-snb4s"] Nov 01 22:44:53.931271 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:44:53.928156 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod212e6260_16ba_4d02_9f5d_84cf58050476.slice/crio-79f935dd3ee2969fbc5d779836056e537e12ba630184dad6754b3ae00f97f5f6 WatchSource:0}: Error finding container 79f935dd3ee2969fbc5d779836056e537e12ba630184dad6754b3ae00f97f5f6: Status 404 returned error can't find the container with id 79f935dd3ee2969fbc5d779836056e537e12ba630184dad6754b3ae00f97f5f6 Nov 01 22:44:54.020512 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:44:54.020476 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda187f76d_d133_49a8_9614_6dcd3fe2f1b0.slice/crio-da49b047282cfe68b7b8436079ea5014098bcac5f7897dbebcec28762797c62c WatchSource:0}: Error finding container da49b047282cfe68b7b8436079ea5014098bcac5f7897dbebcec28762797c62c: Status 404 returned error can't find the container with id da49b047282cfe68b7b8436079ea5014098bcac5f7897dbebcec28762797c62c Nov 01 22:44:54.021131 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:54.020996 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-56b833-d86bfc667-g45bg"] Nov 01 22:44:54.412684 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:54.412631 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56b833-556fbfdf9-snb4s" event={"ID":"212e6260-16ba-4d02-9f5d-84cf58050476","Type":"ContainerStarted","Data":"79f935dd3ee2969fbc5d779836056e537e12ba630184dad6754b3ae00f97f5f6"} Nov 01 22:44:54.484420 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:54.484334 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56b833-d86bfc667-g45bg" event={"ID":"a187f76d-d133-49a8-9614-6dcd3fe2f1b0","Type":"ContainerStarted","Data":"da49b047282cfe68b7b8436079ea5014098bcac5f7897dbebcec28762797c62c"} Nov 01 22:44:55.528062 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:55.528022 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56b833-556fbfdf9-snb4s" event={"ID":"212e6260-16ba-4d02-9f5d-84cf58050476","Type":"ContainerStarted","Data":"acd95f134942f10f6aac371c1e1d03a5e5e865bb85c0a2cecfe2de4430bf5783"} Nov 01 22:44:55.532043 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:55.532016 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56b833-556fbfdf9-snb4s" Nov 01 22:44:55.554867 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:55.552695 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56b833-d86bfc667-g45bg" event={"ID":"a187f76d-d133-49a8-9614-6dcd3fe2f1b0","Type":"ContainerStarted","Data":"106ffa4d58adba88e2d91f7b086247d94860c07449a4a7dc5e29508f28902ed2"} Nov 01 22:44:55.554867 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:55.553765 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56b833-d86bfc667-g45bg" Nov 01 22:44:55.638321 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:55.638251 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56b833-d86bfc667-g45bg" podStartSLOduration=3.6382281340000002 podStartE2EDuration="3.638228134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:55.625097393 +0000 UTC m=+1264.029688029" watchObservedRunningTime="2026-11-01 22:44:55.638228134 +0000 UTC m=+1264.042818771" Nov 01 22:44:55.667861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:55.651154 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56b833-556fbfdf9-snb4s" podStartSLOduration=3.651136286 podStartE2EDuration="3.651136286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:55.596855945 +0000 UTC m=+1264.001446578" watchObservedRunningTime="2026-11-01 22:44:55.651136286 +0000 UTC m=+1264.055726922" Nov 01 22:44:56.146939 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:56.136902 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:56.146939 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:56.145927 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jkvvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:44:56.146939 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:56.145991 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" podUID="1b6bec10-9ac2-4708-9631-b76c801c0c58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:56.565857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:56.559900 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4534ad-666786f5b7-t2kpt" podUID="22dc49f2-d19a-4ff8-957f-e333317212a8" containerName="payload-pre-processing" containerID="cri-o://d8f3cc2c5f48313fd5fd26fbd0f7ddc86585890ee504decea1d58277f5ac215e" gracePeriod=25 Nov 01 22:44:56.759860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:56.748892 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:56.842601 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:56.840241 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:57.025608 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:57.025559 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:44:57.063281 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:57.036065 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 22:44:57.300005 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:57.292942 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:57.300005 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:57.293404 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:44:57.764206 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:57.764149 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:57.788408 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:57.780071 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:44:57.883934 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:57.869955 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 22:44:57.883934 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:57.870069 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 22:44:57.883934 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:57.871296 5011 generic.go:356] "Generic (PLEG): container finished" podID="22dc49f2-d19a-4ff8-957f-e333317212a8" containerID="d8f3cc2c5f48313fd5fd26fbd0f7ddc86585890ee504decea1d58277f5ac215e" exitCode=0 Nov 01 22:44:57.883934 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:57.872372 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4534ad-666786f5b7-t2kpt" event={"ID":"22dc49f2-d19a-4ff8-957f-e333317212a8","Type":"ContainerDied","Data":"d8f3cc2c5f48313fd5fd26fbd0f7ddc86585890ee504decea1d58277f5ac215e"} Nov 01 22:44:57.883934 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:57.877164 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4534ad-666786f5b7-t2kpt" Nov 01 22:44:57.994779 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:57.994689 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22dc49f2-d19a-4ff8-957f-e333317212a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22dc49f2-d19a-4ff8-957f-e333317212a8-plugins-config-volume\") pod \"22dc49f2-d19a-4ff8-957f-e333317212a8\" (UID: \"22dc49f2-d19a-4ff8-957f-e333317212a8\") " Nov 01 22:44:57.995384 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:57.995353 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22dc49f2-d19a-4ff8-957f-e333317212a8-kube-api-access-cv2q4\" (UniqueName: \"kubernetes.io/projected/22dc49f2-d19a-4ff8-957f-e333317212a8-kube-api-access-cv2q4\") pod \"22dc49f2-d19a-4ff8-957f-e333317212a8\" (UID: \"22dc49f2-d19a-4ff8-957f-e333317212a8\") " Nov 01 22:44:57.999436 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:57.999406 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22dc49f2-d19a-4ff8-957f-e333317212a8-plugins-config-volume" pod "22dc49f2-d19a-4ff8-957f-e333317212a8" (UID: "22dc49f2-d19a-4ff8-957f-e333317212a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:58.024895 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:58.024857 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22dc49f2-d19a-4ff8-957f-e333317212a8-kube-api-access-cv2q4" pod "22dc49f2-d19a-4ff8-957f-e333317212a8" (UID: "22dc49f2-d19a-4ff8-957f-e333317212a8"). InnerVolumeSpecName "kube-api-access-cv2q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:58.107931 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:58.099430 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv2q4\" (UniqueName: \"kubernetes.io/projected/22dc49f2-d19a-4ff8-957f-e333317212a8-kube-api-access-cv2q4\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:58.107931 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:58.099489 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22dc49f2-d19a-4ff8-957f-e333317212a8-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:58.958863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:58.956100 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4534ad-666786f5b7-t2kpt" Nov 01 22:44:58.988090 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:58.988031 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4534ad-666786f5b7-t2kpt" event={"ID":"22dc49f2-d19a-4ff8-957f-e333317212a8","Type":"ContainerDied","Data":"06574b0824ad25ff3ddde4e9bbf73880799911435c53e84012d58e8508a2d45d"} Nov 01 22:44:58.988375 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:58.988356 5011 scope.go:122] "RemoveContainer" containerID="d8f3cc2c5f48313fd5fd26fbd0f7ddc86585890ee504decea1d58277f5ac215e" Nov 01 22:44:59.090550 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:59.090522 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4534ad-666786f5b7-t2kpt"] Nov 01 22:44:59.097092 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:59.097062 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4534ad-666786f5b7-t2kpt"] Nov 01 22:44:59.753960 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:44:59.753924 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22dc49f2-d19a-4ff8-957f-e333317212a8" path="/var/lib/kubelet/pods/22dc49f2-d19a-4ff8-957f-e333317212a8/volumes" Nov 01 22:45:00.292566 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.292526 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892885-sqfmb"] Nov 01 22:45:00.293804 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.293774 5011 cpu_manager.go:414] "containerMap: removing container" podUID="22dc49f2-d19a-4ff8-957f-e333317212a8" containerName="payload-pre-processing" Nov 01 22:45:00.293804 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.293804 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22dc49f2-d19a-4ff8-957f-e333317212a8" containerName="payload-pre-processing" Nov 01 22:45:00.294068 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.294052 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="22dc49f2-d19a-4ff8-957f-e333317212a8" containerName="payload-pre-processing" Nov 01 22:45:00.300018 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.299990 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892885-sqfmb" Nov 01 22:45:00.316128 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.316093 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:45:00.320734 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.320246 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892885-sqfmb"] Nov 01 22:45:00.358958 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.358915 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892885-8gfsk"] Nov 01 22:45:00.365155 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.365132 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-8gfsk" Nov 01 22:45:00.365457 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.365434 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p"] Nov 01 22:45:00.374110 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.374079 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p" Nov 01 22:45:00.402197 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.402159 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l9qd\" (UniqueName: \"kubernetes.io/projected/600fad75-5eab-47e7-b929-a5cd12963bf0-kube-api-access-6l9qd\") pod \"maas-api-key-cleanup-e2e-worker-w2-9ad303-29892885-sqfmb\" (UID: \"600fad75-5eab-47e7-b929-a5cd12963bf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892885-sqfmb" Nov 01 22:45:00.402374 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.402239 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwzst\" (UniqueName: \"kubernetes.io/projected/7f8111ee-a594-407a-92ee-48fa084f0fd1-kube-api-access-pwzst\") pod \"maas-api-key-cleanup-29892885-8gfsk\" (UID: \"7f8111ee-a594-407a-92ee-48fa084f0fd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-8gfsk" Nov 01 22:45:00.402374 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.402295 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46f12a8-5d2a-4265-92ac-dab15dbe095d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p\" (UID: \"b46f12a8-5d2a-4265-92ac-dab15dbe095d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p" Nov 01 22:45:00.402374 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.402324 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f8111ee-a594-407a-92ee-48fa084f0fd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892885-8gfsk\" (UID: \"7f8111ee-a594-407a-92ee-48fa084f0fd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-8gfsk" Nov 01 22:45:00.402593 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.402529 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/600fad75-5eab-47e7-b929-a5cd12963bf0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9ad303-29892885-sqfmb\" (UID: \"600fad75-5eab-47e7-b929-a5cd12963bf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892885-sqfmb" Nov 01 22:45:00.402760 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.402737 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djr46\" (UniqueName: \"kubernetes.io/projected/b46f12a8-5d2a-4265-92ac-dab15dbe095d-kube-api-access-djr46\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p\" (UID: \"b46f12a8-5d2a-4265-92ac-dab15dbe095d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p" Nov 01 22:45:00.403005 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.402956 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p"] Nov 01 22:45:00.415941 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.411939 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892885-8gfsk"] Nov 01 22:45:00.422007 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.420999 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75"] Nov 01 22:45:00.428520 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.428480 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75"] Nov 01 22:45:00.428683 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.428611 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75" Nov 01 22:45:00.504402 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.504367 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/600fad75-5eab-47e7-b929-a5cd12963bf0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9ad303-29892885-sqfmb\" (UID: \"600fad75-5eab-47e7-b929-a5cd12963bf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892885-sqfmb" Nov 01 22:45:00.504598 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.504492 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djr46\" (UniqueName: \"kubernetes.io/projected/b46f12a8-5d2a-4265-92ac-dab15dbe095d-kube-api-access-djr46\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p\" (UID: \"b46f12a8-5d2a-4265-92ac-dab15dbe095d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p" Nov 01 22:45:00.504598 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.504566 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l9qd\" (UniqueName: \"kubernetes.io/projected/600fad75-5eab-47e7-b929-a5cd12963bf0-kube-api-access-6l9qd\") pod \"maas-api-key-cleanup-e2e-worker-w2-9ad303-29892885-sqfmb\" (UID: \"600fad75-5eab-47e7-b929-a5cd12963bf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892885-sqfmb" Nov 01 22:45:00.504732 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.504607 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwzst\" (UniqueName: \"kubernetes.io/projected/7f8111ee-a594-407a-92ee-48fa084f0fd1-kube-api-access-pwzst\") pod \"maas-api-key-cleanup-29892885-8gfsk\" (UID: \"7f8111ee-a594-407a-92ee-48fa084f0fd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-8gfsk" Nov 01 22:45:00.504732 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.504654 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46f12a8-5d2a-4265-92ac-dab15dbe095d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p\" (UID: \"b46f12a8-5d2a-4265-92ac-dab15dbe095d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p" Nov 01 22:45:00.504732 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.504680 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f8111ee-a594-407a-92ee-48fa084f0fd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892885-8gfsk\" (UID: \"7f8111ee-a594-407a-92ee-48fa084f0fd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-8gfsk" Nov 01 22:45:00.504934 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.504762 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbplh\" (UniqueName: \"kubernetes.io/projected/0b6a3491-66c7-42a2-b43b-307bf20f52d1-kube-api-access-zbplh\") pod \"maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75\" (UID: \"0b6a3491-66c7-42a2-b43b-307bf20f52d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75" Nov 01 22:45:00.504934 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.504793 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6a3491-66c7-42a2-b43b-307bf20f52d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75\" (UID: \"0b6a3491-66c7-42a2-b43b-307bf20f52d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75" Nov 01 22:45:00.505214 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.505192 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/600fad75-5eab-47e7-b929-a5cd12963bf0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9ad303-29892885-sqfmb\" (UID: \"600fad75-5eab-47e7-b929-a5cd12963bf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892885-sqfmb" Nov 01 22:45:00.505376 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.505357 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f8111ee-a594-407a-92ee-48fa084f0fd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892885-8gfsk\" (UID: \"7f8111ee-a594-407a-92ee-48fa084f0fd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-8gfsk" Nov 01 22:45:00.505415 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.505393 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46f12a8-5d2a-4265-92ac-dab15dbe095d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p\" (UID: \"b46f12a8-5d2a-4265-92ac-dab15dbe095d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p" Nov 01 22:45:00.517007 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.516813 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l9qd\" (UniqueName: \"kubernetes.io/projected/600fad75-5eab-47e7-b929-a5cd12963bf0-kube-api-access-6l9qd\") pod \"maas-api-key-cleanup-e2e-worker-w2-9ad303-29892885-sqfmb\" (UID: \"600fad75-5eab-47e7-b929-a5cd12963bf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892885-sqfmb" Nov 01 22:45:00.517953 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.517925 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djr46\" (UniqueName: \"kubernetes.io/projected/b46f12a8-5d2a-4265-92ac-dab15dbe095d-kube-api-access-djr46\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p\" (UID: \"b46f12a8-5d2a-4265-92ac-dab15dbe095d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p" Nov 01 22:45:00.521554 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.521526 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwzst\" (UniqueName: \"kubernetes.io/projected/7f8111ee-a594-407a-92ee-48fa084f0fd1-kube-api-access-pwzst\") pod \"maas-api-key-cleanup-29892885-8gfsk\" (UID: \"7f8111ee-a594-407a-92ee-48fa084f0fd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-8gfsk" Nov 01 22:45:00.606896 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.606778 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbplh\" (UniqueName: \"kubernetes.io/projected/0b6a3491-66c7-42a2-b43b-307bf20f52d1-kube-api-access-zbplh\") pod \"maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75\" (UID: \"0b6a3491-66c7-42a2-b43b-307bf20f52d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75" Nov 01 22:45:00.606896 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.606855 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6a3491-66c7-42a2-b43b-307bf20f52d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75\" (UID: \"0b6a3491-66c7-42a2-b43b-307bf20f52d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75" Nov 01 22:45:00.607705 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.607681 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6a3491-66c7-42a2-b43b-307bf20f52d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75\" (UID: \"0b6a3491-66c7-42a2-b43b-307bf20f52d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75" Nov 01 22:45:00.617295 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.617259 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbplh\" (UniqueName: \"kubernetes.io/projected/0b6a3491-66c7-42a2-b43b-307bf20f52d1-kube-api-access-zbplh\") pod \"maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75\" (UID: \"0b6a3491-66c7-42a2-b43b-307bf20f52d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75" Nov 01 22:45:00.632876 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.632845 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892885-sqfmb" Nov 01 22:45:00.700901 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.700696 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-8gfsk" Nov 01 22:45:00.726192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.726152 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p" Nov 01 22:45:00.754734 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.754479 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75" Nov 01 22:45:00.933889 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:00.933707 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892885-sqfmb"] Nov 01 22:45:01.022893 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:01.006441 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892885-sqfmb" event={"ID":"600fad75-5eab-47e7-b929-a5cd12963bf0","Type":"ContainerStarted","Data":"e307fe89667745ccecebe333cd2d734251f31e4ad363d50bb56355793384b065"} Nov 01 22:45:01.046388 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:45:01.046346 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f8111ee_a594_407a_92ee_48fa084f0fd1.slice/crio-c2d8d24cad97299fc2834f496e3cbbfb986cf0100cc09e43ffb18c23e6e2b37c WatchSource:0}: Error finding container c2d8d24cad97299fc2834f496e3cbbfb986cf0100cc09e43ffb18c23e6e2b37c: Status 404 returned error can't find the container with id c2d8d24cad97299fc2834f496e3cbbfb986cf0100cc09e43ffb18c23e6e2b37c Nov 01 22:45:01.055574 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:01.055532 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892885-8gfsk"] Nov 01 22:45:01.093680 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:45:01.093648 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb46f12a8_5d2a_4265_92ac_dab15dbe095d.slice/crio-ccfcfdc66e0dd26e59a1b99ee0549b61e3db767ef61ceae8a51e38644d4ba1a7 WatchSource:0}: Error finding container ccfcfdc66e0dd26e59a1b99ee0549b61e3db767ef61ceae8a51e38644d4ba1a7: Status 404 returned error can't find the container with id ccfcfdc66e0dd26e59a1b99ee0549b61e3db767ef61ceae8a51e38644d4ba1a7 Nov 01 22:45:01.095534 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:01.095509 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p"] Nov 01 22:45:01.105032 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:01.105005 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75"] Nov 01 22:45:01.108568 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:45:01.108534 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6a3491_66c7_42a2_b43b_307bf20f52d1.slice/crio-d0ec851426990deaf5ab4003354f0020ab8ba00d38a6f6aeb780a6cad7b641cf WatchSource:0}: Error finding container d0ec851426990deaf5ab4003354f0020ab8ba00d38a6f6aeb780a6cad7b641cf: Status 404 returned error can't find the container with id d0ec851426990deaf5ab4003354f0020ab8ba00d38a6f6aeb780a6cad7b641cf Nov 01 22:45:02.028162 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:02.028118 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75" event={"ID":"0b6a3491-66c7-42a2-b43b-307bf20f52d1","Type":"ContainerStarted","Data":"d0ec851426990deaf5ab4003354f0020ab8ba00d38a6f6aeb780a6cad7b641cf"} Nov 01 22:45:02.031498 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:02.031458 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-8gfsk" event={"ID":"7f8111ee-a594-407a-92ee-48fa084f0fd1","Type":"ContainerStarted","Data":"c2d8d24cad97299fc2834f496e3cbbfb986cf0100cc09e43ffb18c23e6e2b37c"} Nov 01 22:45:02.034417 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:02.034383 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p" event={"ID":"b46f12a8-5d2a-4265-92ac-dab15dbe095d","Type":"ContainerStarted","Data":"ccfcfdc66e0dd26e59a1b99ee0549b61e3db767ef61ceae8a51e38644d4ba1a7"} Nov 01 22:45:07.094656 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:07.094568 5011 generic.go:356] "Generic (PLEG): container finished" podID="58c1620e-7736-4fe3-bfde-8024cfb8d2e5" containerID="dcce2c0ba8383a16bd9638aa080632c45f034cb3bc7780fbb9fcbd47213854f5" exitCode=0 Nov 01 22:45:07.095142 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:07.094646 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" event={"ID":"58c1620e-7736-4fe3-bfde-8024cfb8d2e5","Type":"ContainerDied","Data":"dcce2c0ba8383a16bd9638aa080632c45f034cb3bc7780fbb9fcbd47213854f5"} Nov 01 22:45:07.095209 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:07.095192 5011 scope.go:122] "RemoveContainer" containerID="dcce2c0ba8383a16bd9638aa080632c45f034cb3bc7780fbb9fcbd47213854f5" Nov 01 22:45:07.131887 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:07.131851 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:45:07.132039 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:07.131923 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:45:07.132039 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:07.131955 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:45:07.876878 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:07.876806 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56b833-556fbfdf9-snb4s" Nov 01 22:45:07.877044 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:07.876919 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56b833-d86bfc667-g45bg" Nov 01 22:45:08.106022 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:08.105990 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892885-sqfmb" event={"ID":"600fad75-5eab-47e7-b929-a5cd12963bf0","Type":"ContainerStarted","Data":"0f3de00f167d9b7f7e9d2887690baebea59a7ab7b217dcc79fe2d97a8d76dcf4"} Nov 01 22:45:08.108479 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:08.108450 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" event={"ID":"58c1620e-7736-4fe3-bfde-8024cfb8d2e5","Type":"ContainerStarted","Data":"52f591d9f3e3a4645a5195318b13b1b4272e64b6b0f5785d56cc7e5fc93e1747"} Nov 01 22:45:08.110297 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:08.110274 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75" event={"ID":"0b6a3491-66c7-42a2-b43b-307bf20f52d1","Type":"ContainerStarted","Data":"d92d90eb1467f45f207a5d1c57624542fbdff37e09fd161d0e54ee3b5608cff2"} Nov 01 22:45:08.111882 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:08.111857 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-8gfsk" event={"ID":"7f8111ee-a594-407a-92ee-48fa084f0fd1","Type":"ContainerStarted","Data":"6d4e0dda65ba4ce2bc4e6f074efea8162d7e7494e6a25902bba54d1a3deda542"} Nov 01 22:45:08.113451 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:08.113431 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p" event={"ID":"b46f12a8-5d2a-4265-92ac-dab15dbe095d","Type":"ContainerStarted","Data":"ab3c8ef42d49f4965499a8735cea151797aa2927ee2662f83c6a218508cf11ae"} Nov 01 22:45:08.129815 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:08.129754 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892885-sqfmb" podStartSLOduration=1.82347769 podStartE2EDuration="8.129735122s" totalImagesPullingTime="6.306257432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:08.128617722 +0000 UTC m=+1276.533208354" watchObservedRunningTime="2026-11-01 22:45:08.129735122 +0000 UTC m=+1276.534325755" Nov 01 22:45:08.161025 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:08.160733 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-8gfsk" podStartSLOduration=1.9797590600000001 podStartE2EDuration="8.160716333s" totalImagesPullingTime="6.180957273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:08.158465945 +0000 UTC m=+1276.563056605" watchObservedRunningTime="2026-11-01 22:45:08.160716333 +0000 UTC m=+1276.565306967" Nov 01 22:45:08.209077 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:08.208969 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p" podStartSLOduration=2.147136775 podStartE2EDuration="8.208953681s" totalImagesPullingTime="6.061816906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:08.206500154 +0000 UTC m=+1276.611090787" watchObservedRunningTime="2026-11-01 22:45:08.208953681 +0000 UTC m=+1276.613544313" Nov 01 22:45:08.228042 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:08.227974 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75" podStartSLOduration=2.108346507 podStartE2EDuration="8.227954109s" totalImagesPullingTime="6.119607602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:08.225901536 +0000 UTC m=+1276.630492181" watchObservedRunningTime="2026-11-01 22:45:08.227954109 +0000 UTC m=+1276.632544752" Nov 01 22:45:08.451938 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:08.451903 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:45:08.452901 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:08.452878 5011 patch_prober.go:29] interesting pod/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 01 22:45:08.453014 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:08.452924 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" podUID="58c1620e-7736-4fe3-bfde-8024cfb8d2e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 01 22:45:09.114816 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:09.114735 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6"] Nov 01 22:45:09.125543 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:09.123467 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6" Nov 01 22:45:09.131322 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:09.129679 5011 generic.go:356] "Generic (PLEG): container finished" podID="0b6a3491-66c7-42a2-b43b-307bf20f52d1" containerID="d92d90eb1467f45f207a5d1c57624542fbdff37e09fd161d0e54ee3b5608cff2" exitCode=60 Nov 01 22:45:09.131322 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:09.129784 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75" event={"ID":"0b6a3491-66c7-42a2-b43b-307bf20f52d1","Type":"ContainerDied","Data":"d92d90eb1467f45f207a5d1c57624542fbdff37e09fd161d0e54ee3b5608cff2"} Nov 01 22:45:09.131322 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:09.130283 5011 scope.go:122] "RemoveContainer" containerID="d92d90eb1467f45f207a5d1c57624542fbdff37e09fd161d0e54ee3b5608cff2" Nov 01 22:45:09.139660 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:09.137758 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-56b833-serving-cert\"" Nov 01 22:45:09.139660 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:09.138656 5011 generic.go:356] "Generic (PLEG): container finished" podID="7f8111ee-a594-407a-92ee-48fa084f0fd1" containerID="6d4e0dda65ba4ce2bc4e6f074efea8162d7e7494e6a25902bba54d1a3deda542" exitCode=60 Nov 01 22:45:09.139660 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:09.138883 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-8gfsk" event={"ID":"7f8111ee-a594-407a-92ee-48fa084f0fd1","Type":"ContainerDied","Data":"6d4e0dda65ba4ce2bc4e6f074efea8162d7e7494e6a25902bba54d1a3deda542"} Nov 01 22:45:09.151778 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:09.140369 5011 scope.go:122] "RemoveContainer" containerID="6d4e0dda65ba4ce2bc4e6f074efea8162d7e7494e6a25902bba54d1a3deda542" Nov 01 22:45:09.176355 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:09.176118 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6"] Nov 01 22:45:09.183998 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:09.183957 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tkns\" (UniqueName: \"kubernetes.io/projected/ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb-kube-api-access-2tkns\") pod \"maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6\" (UID: \"ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6" Nov 01 22:45:09.184404 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:09.184382 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6\" (UID: \"ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6" Nov 01 22:45:09.184489 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:09.184471 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6\" (UID: \"ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6" Nov 01 22:45:09.184544 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:09.184511 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6\" (UID: \"ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6" Nov 01 22:45:09.191105 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:09.191069 5011 generic.go:356] "Generic (PLEG): container finished" podID="b46f12a8-5d2a-4265-92ac-dab15dbe095d" containerID="ab3c8ef42d49f4965499a8735cea151797aa2927ee2662f83c6a218508cf11ae" exitCode=60 Nov 01 22:45:09.191292 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:09.191271 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p" event={"ID":"b46f12a8-5d2a-4265-92ac-dab15dbe095d","Type":"ContainerDied","Data":"ab3c8ef42d49f4965499a8735cea151797aa2927ee2662f83c6a218508cf11ae"} Nov 01 22:45:09.191742 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:09.191721 5011 scope.go:122] "RemoveContainer" containerID="ab3c8ef42d49f4965499a8735cea151797aa2927ee2662f83c6a218508cf11ae" Nov 01 22:45:09.203885 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:09.199929 5011 generic.go:356] "Generic (PLEG): container finished" podID="600fad75-5eab-47e7-b929-a5cd12963bf0" containerID="0f3de00f167d9b7f7e9d2887690baebea59a7ab7b217dcc79fe2d97a8d76dcf4" exitCode=60 Nov 01 22:45:09.203885 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:09.200678 5011 scope.go:122] "RemoveContainer" containerID="0f3de00f167d9b7f7e9d2887690baebea59a7ab7b217dcc79fe2d97a8d76dcf4" Nov 01 22:45:09.203885 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:09.200762 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892885-sqfmb" event={"ID":"600fad75-5eab-47e7-b929-a5cd12963bf0","Type":"ContainerDied","Data":"0f3de00f167d9b7f7e9d2887690baebea59a7ab7b217dcc79fe2d97a8d76dcf4"} Nov 01 22:45:09.352858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:09.320240 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6\" (UID: \"ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6" Nov 01 22:45:09.360865 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:09.353159 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6\" (UID: \"ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6" Nov 01 22:45:09.451910 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:09.391867 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6\" (UID: \"ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6" Nov 01 22:45:09.489858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:09.452313 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tkns\" (UniqueName: \"kubernetes.io/projected/ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb-kube-api-access-2tkns\") pod \"maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6\" (UID: \"ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6" Nov 01 22:45:09.494879 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:09.368071 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6\" (UID: \"ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6" Nov 01 22:45:09.495089 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:09.343207 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6\" (UID: \"ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6" Nov 01 22:45:09.501917 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:09.473685 5011 patch_prober.go:29] interesting pod/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 01 22:45:09.503454 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:09.491688 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6\" (UID: \"ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6" Nov 01 22:45:09.503454 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:09.502933 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" podUID="58c1620e-7736-4fe3-bfde-8024cfb8d2e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 01 22:45:09.508512 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:09.508483 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tkns\" (UniqueName: \"kubernetes.io/projected/ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb-kube-api-access-2tkns\") pod \"maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6\" (UID: \"ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6" Nov 01 22:45:09.755431 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:09.755348 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6" Nov 01 22:45:10.494533 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:10.461136 5011 patch_prober.go:29] interesting pod/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 01 22:45:10.494533 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:10.461193 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" podUID="58c1620e-7736-4fe3-bfde-8024cfb8d2e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 01 22:45:10.608454 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:10.601888 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75" event={"ID":"0b6a3491-66c7-42a2-b43b-307bf20f52d1","Type":"ContainerStarted","Data":"efe294adaa78c05c22c45c116c1813af024ee145846cf3c1352925e286960d35"} Nov 01 22:45:10.608454 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:10.607958 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p" event={"ID":"b46f12a8-5d2a-4265-92ac-dab15dbe095d","Type":"ContainerStarted","Data":"d601d64ec850dd57b9ad1ff41cf8fa21ab873ccc922a8f6f21fb62497bbfc831"} Nov 01 22:45:10.710044 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:45:10.710003 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad981a0b_7ed9_4f4f_acc4_3009d4ca38bb.slice/crio-bb61d030c60135e205eee1b4c77eef5b6f5e45ab86738ea340c9305c0851cd79 WatchSource:0}: Error finding container bb61d030c60135e205eee1b4c77eef5b6f5e45ab86738ea340c9305c0851cd79: Status 404 returned error can't find the container with id bb61d030c60135e205eee1b4c77eef5b6f5e45ab86738ea340c9305c0851cd79 Nov 01 22:45:10.710603 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:10.710556 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6"] Nov 01 22:45:10.831009 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:10.830971 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:45:11.460867 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:11.460254 5011 patch_prober.go:29] interesting pod/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 01 22:45:11.460867 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:11.460316 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" podUID="58c1620e-7736-4fe3-bfde-8024cfb8d2e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 01 22:45:11.620889 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:11.620769 5011 generic.go:356] "Generic (PLEG): container finished" podID="b46f12a8-5d2a-4265-92ac-dab15dbe095d" containerID="d601d64ec850dd57b9ad1ff41cf8fa21ab873ccc922a8f6f21fb62497bbfc831" exitCode=60 Nov 01 22:45:11.620889 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:11.620858 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p" event={"ID":"b46f12a8-5d2a-4265-92ac-dab15dbe095d","Type":"ContainerDied","Data":"d601d64ec850dd57b9ad1ff41cf8fa21ab873ccc922a8f6f21fb62497bbfc831"} Nov 01 22:45:11.621519 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:11.620927 5011 scope.go:122] "RemoveContainer" containerID="ab3c8ef42d49f4965499a8735cea151797aa2927ee2662f83c6a218508cf11ae" Nov 01 22:45:11.621519 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:11.621314 5011 scope.go:122] "RemoveContainer" containerID="d601d64ec850dd57b9ad1ff41cf8fa21ab873ccc922a8f6f21fb62497bbfc831" Nov 01 22:45:11.621726 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:45:11.621698 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p_odh-ai-gateway-infra(b46f12a8-5d2a-4265-92ac-dab15dbe095d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p" podUID="b46f12a8-5d2a-4265-92ac-dab15dbe095d" Nov 01 22:45:11.623911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:11.623883 5011 generic.go:356] "Generic (PLEG): container finished" podID="600fad75-5eab-47e7-b929-a5cd12963bf0" containerID="599f79892e5e6c2203d22d746bd61f44b0e6e41b135ba377083601cea303140e" exitCode=60 Nov 01 22:45:11.624024 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:11.623974 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892885-sqfmb" event={"ID":"600fad75-5eab-47e7-b929-a5cd12963bf0","Type":"ContainerDied","Data":"599f79892e5e6c2203d22d746bd61f44b0e6e41b135ba377083601cea303140e"} Nov 01 22:45:11.624410 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:11.624391 5011 scope.go:122] "RemoveContainer" containerID="599f79892e5e6c2203d22d746bd61f44b0e6e41b135ba377083601cea303140e" Nov 01 22:45:11.624776 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:45:11.624756 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9ad303-29892885-sqfmb_odh-ai-gateway-infra(600fad75-5eab-47e7-b929-a5cd12963bf0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892885-sqfmb" podUID="600fad75-5eab-47e7-b929-a5cd12963bf0" Nov 01 22:45:11.626017 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:11.625991 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6" event={"ID":"ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb","Type":"ContainerStarted","Data":"37a1f61c7855eed2278050904d42015fc3c5686cae82fc42f7be097788da353b"} Nov 01 22:45:11.626167 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:11.626042 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6" event={"ID":"ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb","Type":"ContainerStarted","Data":"bb61d030c60135e205eee1b4c77eef5b6f5e45ab86738ea340c9305c0851cd79"} Nov 01 22:45:11.626167 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:11.626073 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6" Nov 01 22:45:11.629618 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:11.629479 5011 generic.go:356] "Generic (PLEG): container finished" podID="0b6a3491-66c7-42a2-b43b-307bf20f52d1" containerID="efe294adaa78c05c22c45c116c1813af024ee145846cf3c1352925e286960d35" exitCode=60 Nov 01 22:45:11.629618 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:11.629543 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75" event={"ID":"0b6a3491-66c7-42a2-b43b-307bf20f52d1","Type":"ContainerDied","Data":"efe294adaa78c05c22c45c116c1813af024ee145846cf3c1352925e286960d35"} Nov 01 22:45:11.629879 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:11.629861 5011 scope.go:122] "RemoveContainer" containerID="efe294adaa78c05c22c45c116c1813af024ee145846cf3c1352925e286960d35" Nov 01 22:45:11.630237 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:45:11.630210 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75_odh-ai-gateway-infra(0b6a3491-66c7-42a2-b43b-307bf20f52d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75" podUID="0b6a3491-66c7-42a2-b43b-307bf20f52d1" Nov 01 22:45:11.632077 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:11.632053 5011 generic.go:356] "Generic (PLEG): container finished" podID="7f8111ee-a594-407a-92ee-48fa084f0fd1" containerID="bc9b9d886f53f1cd5eb17233d7fc3c58e6facff5f96c97efc8370338f678bca6" exitCode=60 Nov 01 22:45:11.632170 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:11.632091 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-8gfsk" event={"ID":"7f8111ee-a594-407a-92ee-48fa084f0fd1","Type":"ContainerDied","Data":"bc9b9d886f53f1cd5eb17233d7fc3c58e6facff5f96c97efc8370338f678bca6"} Nov 01 22:45:11.632519 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:11.632500 5011 scope.go:122] "RemoveContainer" containerID="bc9b9d886f53f1cd5eb17233d7fc3c58e6facff5f96c97efc8370338f678bca6" Nov 01 22:45:11.632891 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:45:11.632866 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892885-8gfsk_odh-ai-gateway-infra(7f8111ee-a594-407a-92ee-48fa084f0fd1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-8gfsk" podUID="7f8111ee-a594-407a-92ee-48fa084f0fd1" Nov 01 22:45:11.637195 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:11.637175 5011 scope.go:122] "RemoveContainer" containerID="0f3de00f167d9b7f7e9d2887690baebea59a7ab7b217dcc79fe2d97a8d76dcf4" Nov 01 22:45:11.682088 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:11.682065 5011 scope.go:122] "RemoveContainer" containerID="d92d90eb1467f45f207a5d1c57624542fbdff37e09fd161d0e54ee3b5608cff2" Nov 01 22:45:11.697362 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:11.697341 5011 scope.go:122] "RemoveContainer" containerID="6d4e0dda65ba4ce2bc4e6f074efea8162d7e7494e6a25902bba54d1a3deda542" Nov 01 22:45:11.740885 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:11.734408 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6" podStartSLOduration=2.177302435 podStartE2EDuration="2.734386534s" totalImagesPullingTime="557.084099ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:11.72749597 +0000 UTC m=+1280.132086597" watchObservedRunningTime="2026-11-01 22:45:11.734386534 +0000 UTC m=+1280.138977168" Nov 01 22:45:12.462856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:12.462271 5011 patch_prober.go:29] interesting pod/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 01 22:45:12.462856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:12.462334 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" podUID="58c1620e-7736-4fe3-bfde-8024cfb8d2e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 01 22:45:12.670774 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:12.670740 5011 scope.go:122] "RemoveContainer" containerID="efe294adaa78c05c22c45c116c1813af024ee145846cf3c1352925e286960d35" Nov 01 22:45:12.671290 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:45:12.671215 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75_odh-ai-gateway-infra(0b6a3491-66c7-42a2-b43b-307bf20f52d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75" podUID="0b6a3491-66c7-42a2-b43b-307bf20f52d1" Nov 01 22:45:12.674336 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:12.674314 5011 scope.go:122] "RemoveContainer" containerID="d601d64ec850dd57b9ad1ff41cf8fa21ab873ccc922a8f6f21fb62497bbfc831" Nov 01 22:45:12.674690 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:45:12.674671 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p_odh-ai-gateway-infra(b46f12a8-5d2a-4265-92ac-dab15dbe095d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p" podUID="b46f12a8-5d2a-4265-92ac-dab15dbe095d" Nov 01 22:45:13.466074 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:13.466024 5011 patch_prober.go:29] interesting pod/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 01 22:45:13.466320 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:13.466108 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" podUID="58c1620e-7736-4fe3-bfde-8024cfb8d2e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 01 22:45:14.461859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:14.461250 5011 patch_prober.go:29] interesting pod/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 01 22:45:14.461859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:14.461320 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" podUID="58c1620e-7736-4fe3-bfde-8024cfb8d2e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 01 22:45:15.463036 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:15.463005 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:45:15.464405 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:15.464368 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:45:16.961228 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:45:16.961193 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:45:17.680867 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:17.679934 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6" Nov 01 22:45:18.085911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:18.085163 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:45:19.212230 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:19.212186 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7"] Nov 01 22:45:19.213167 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:19.213134 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7" podUID="7f568e19-ff42-4413-a06f-34f9f9594077" containerName="maas-api" containerID="cri-o://10f168e05d6bf075acbd6e9b1cf26438447da9bd38521908059267a1dd066e44" gracePeriod=30 Nov 01 22:45:20.172056 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:20.164588 5011 generic.go:356] "Generic (PLEG): container finished" podID="7f568e19-ff42-4413-a06f-34f9f9594077" containerID="10f168e05d6bf075acbd6e9b1cf26438447da9bd38521908059267a1dd066e44" exitCode=0 Nov 01 22:45:20.172056 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:20.164729 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7" event={"ID":"7f568e19-ff42-4413-a06f-34f9f9594077","Type":"ContainerDied","Data":"10f168e05d6bf075acbd6e9b1cf26438447da9bd38521908059267a1dd066e44"} Nov 01 22:45:20.200546 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:20.200515 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7" Nov 01 22:45:20.278110 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:20.278068 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f568e19-ff42-4413-a06f-34f9f9594077-kube-api-access-2rxpm\" (UniqueName: \"kubernetes.io/projected/7f568e19-ff42-4413-a06f-34f9f9594077-kube-api-access-2rxpm\") pod \"7f568e19-ff42-4413-a06f-34f9f9594077\" (UID: \"7f568e19-ff42-4413-a06f-34f9f9594077\") " Nov 01 22:45:20.278589 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:20.278195 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f568e19-ff42-4413-a06f-34f9f9594077-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f568e19-ff42-4413-a06f-34f9f9594077-openshift-service-ca\") pod \"7f568e19-ff42-4413-a06f-34f9f9594077\" (UID: \"7f568e19-ff42-4413-a06f-34f9f9594077\") " Nov 01 22:45:20.278589 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:20.278312 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f568e19-ff42-4413-a06f-34f9f9594077-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f568e19-ff42-4413-a06f-34f9f9594077-maas-api-tls\") pod \"7f568e19-ff42-4413-a06f-34f9f9594077\" (UID: \"7f568e19-ff42-4413-a06f-34f9f9594077\") " Nov 01 22:45:20.278589 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:20.278384 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f568e19-ff42-4413-a06f-34f9f9594077-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f568e19-ff42-4413-a06f-34f9f9594077-metrics-certs\") pod \"7f568e19-ff42-4413-a06f-34f9f9594077\" (UID: \"7f568e19-ff42-4413-a06f-34f9f9594077\") " Nov 01 22:45:20.278763 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:20.278699 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f568e19-ff42-4413-a06f-34f9f9594077-openshift-service-ca" pod "7f568e19-ff42-4413-a06f-34f9f9594077" (UID: "7f568e19-ff42-4413-a06f-34f9f9594077"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:20.279286 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:20.279255 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f568e19-ff42-4413-a06f-34f9f9594077-openshift-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.281705 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:20.281673 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f568e19-ff42-4413-a06f-34f9f9594077-maas-api-tls" pod "7f568e19-ff42-4413-a06f-34f9f9594077" (UID: "7f568e19-ff42-4413-a06f-34f9f9594077"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:20.282219 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:20.282192 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f568e19-ff42-4413-a06f-34f9f9594077-kube-api-access-2rxpm" pod "7f568e19-ff42-4413-a06f-34f9f9594077" (UID: "7f568e19-ff42-4413-a06f-34f9f9594077"). InnerVolumeSpecName "kube-api-access-2rxpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:20.284115 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:20.283337 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f568e19-ff42-4413-a06f-34f9f9594077-metrics-certs" pod "7f568e19-ff42-4413-a06f-34f9f9594077" (UID: "7f568e19-ff42-4413-a06f-34f9f9594077"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:20.380967 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:20.380910 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rxpm\" (UniqueName: \"kubernetes.io/projected/7f568e19-ff42-4413-a06f-34f9f9594077-kube-api-access-2rxpm\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.380967 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:20.380966 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f568e19-ff42-4413-a06f-34f9f9594077-maas-api-tls\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.381205 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:20.380981 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f568e19-ff42-4413-a06f-34f9f9594077-metrics-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:21.173720 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:21.173681 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7" event={"ID":"7f568e19-ff42-4413-a06f-34f9f9594077","Type":"ContainerDied","Data":"fbe61b5593bf1827be5f25f191165d24f101722ee96bcd9f40015846b7d2c84a"} Nov 01 22:45:21.173917 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:21.173753 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7" Nov 01 22:45:21.173917 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:21.173756 5011 scope.go:122] "RemoveContainer" containerID="10f168e05d6bf075acbd6e9b1cf26438447da9bd38521908059267a1dd066e44" Nov 01 22:45:21.225223 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:21.220884 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7"] Nov 01 22:45:21.227872 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:21.227818 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4534ad-75dbc489cb-jqlh7"] Nov 01 22:45:21.746611 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:21.746585 5011 scope.go:122] "RemoveContainer" containerID="bc9b9d886f53f1cd5eb17233d7fc3c58e6facff5f96c97efc8370338f678bca6" Nov 01 22:45:21.747188 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:21.747163 5011 scope.go:122] "RemoveContainer" containerID="d601d64ec850dd57b9ad1ff41cf8fa21ab873ccc922a8f6f21fb62497bbfc831" Nov 01 22:45:21.755194 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:21.752275 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f568e19-ff42-4413-a06f-34f9f9594077" path="/var/lib/kubelet/pods/7f568e19-ff42-4413-a06f-34f9f9594077/volumes" Nov 01 22:45:22.183564 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:22.183531 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p" event={"ID":"b46f12a8-5d2a-4265-92ac-dab15dbe095d","Type":"ContainerStarted","Data":"dd73dd5ec88ec9cb103c59942dfc0a3ca26104fff42826c9e87cb29e559b0e9d"} Nov 01 22:45:22.749061 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:22.747326 5011 scope.go:122] "RemoveContainer" containerID="efe294adaa78c05c22c45c116c1813af024ee145846cf3c1352925e286960d35" Nov 01 22:45:23.195692 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:23.195664 5011 generic.go:356] "Generic (PLEG): container finished" podID="7f8111ee-a594-407a-92ee-48fa084f0fd1" containerID="ea83ed8ee57eeaab03c0c43d1713326333c82fe1fe32f27d5f07ed696948d690" exitCode=60 Nov 01 22:45:23.195899 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:23.195740 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-8gfsk" event={"ID":"7f8111ee-a594-407a-92ee-48fa084f0fd1","Type":"ContainerDied","Data":"ea83ed8ee57eeaab03c0c43d1713326333c82fe1fe32f27d5f07ed696948d690"} Nov 01 22:45:23.195899 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:23.195799 5011 scope.go:122] "RemoveContainer" containerID="bc9b9d886f53f1cd5eb17233d7fc3c58e6facff5f96c97efc8370338f678bca6" Nov 01 22:45:23.196328 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:23.196307 5011 scope.go:122] "RemoveContainer" containerID="ea83ed8ee57eeaab03c0c43d1713326333c82fe1fe32f27d5f07ed696948d690" Nov 01 22:45:23.196776 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:45:23.196752 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892885-8gfsk_odh-ai-gateway-infra(7f8111ee-a594-407a-92ee-48fa084f0fd1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-8gfsk" podUID="7f8111ee-a594-407a-92ee-48fa084f0fd1" Nov 01 22:45:23.198165 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:23.198146 5011 generic.go:356] "Generic (PLEG): container finished" podID="b46f12a8-5d2a-4265-92ac-dab15dbe095d" containerID="dd73dd5ec88ec9cb103c59942dfc0a3ca26104fff42826c9e87cb29e559b0e9d" exitCode=60 Nov 01 22:45:23.198262 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:23.198221 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p" event={"ID":"b46f12a8-5d2a-4265-92ac-dab15dbe095d","Type":"ContainerDied","Data":"dd73dd5ec88ec9cb103c59942dfc0a3ca26104fff42826c9e87cb29e559b0e9d"} Nov 01 22:45:23.198600 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:23.198572 5011 scope.go:122] "RemoveContainer" containerID="dd73dd5ec88ec9cb103c59942dfc0a3ca26104fff42826c9e87cb29e559b0e9d" Nov 01 22:45:23.198980 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:45:23.198952 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p_odh-ai-gateway-infra(b46f12a8-5d2a-4265-92ac-dab15dbe095d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p" podUID="b46f12a8-5d2a-4265-92ac-dab15dbe095d" Nov 01 22:45:23.210449 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:23.210432 5011 scope.go:122] "RemoveContainer" containerID="d601d64ec850dd57b9ad1ff41cf8fa21ab873ccc922a8f6f21fb62497bbfc831" Nov 01 22:45:23.219991 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:23.219966 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p"] Nov 01 22:45:24.220593 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:24.220495 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75" event={"ID":"0b6a3491-66c7-42a2-b43b-307bf20f52d1","Type":"ContainerDied","Data":"8846ed175cd02f25d871fb67ed8955976d8137f450da33d84a119b80d47000dc"} Nov 01 22:45:24.220593 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:24.220589 5011 scope.go:122] "RemoveContainer" containerID="efe294adaa78c05c22c45c116c1813af024ee145846cf3c1352925e286960d35" Nov 01 22:45:24.221143 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:24.220473 5011 generic.go:356] "Generic (PLEG): container finished" podID="0b6a3491-66c7-42a2-b43b-307bf20f52d1" containerID="8846ed175cd02f25d871fb67ed8955976d8137f450da33d84a119b80d47000dc" exitCode=60 Nov 01 22:45:24.221206 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:24.221137 5011 scope.go:122] "RemoveContainer" containerID="8846ed175cd02f25d871fb67ed8955976d8137f450da33d84a119b80d47000dc" Nov 01 22:45:24.221547 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:45:24.221494 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75_odh-ai-gateway-infra(0b6a3491-66c7-42a2-b43b-307bf20f52d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75" podUID="0b6a3491-66c7-42a2-b43b-307bf20f52d1" Nov 01 22:45:24.229809 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:24.229783 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892885-8gfsk"] Nov 01 22:45:24.442946 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:24.442918 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p" Nov 01 22:45:24.529882 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:24.528591 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b46f12a8-5d2a-4265-92ac-dab15dbe095d-kube-api-access-djr46\" (UniqueName: \"kubernetes.io/projected/b46f12a8-5d2a-4265-92ac-dab15dbe095d-kube-api-access-djr46\") pod \"b46f12a8-5d2a-4265-92ac-dab15dbe095d\" (UID: \"b46f12a8-5d2a-4265-92ac-dab15dbe095d\") " Nov 01 22:45:24.534485 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:24.534447 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b46f12a8-5d2a-4265-92ac-dab15dbe095d-kube-api-access-djr46" pod "b46f12a8-5d2a-4265-92ac-dab15dbe095d" (UID: "b46f12a8-5d2a-4265-92ac-dab15dbe095d"). InnerVolumeSpecName "kube-api-access-djr46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:24.535504 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:24.535392 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b46f12a8-5d2a-4265-92ac-dab15dbe095d-maas-api-service-ca" pod "b46f12a8-5d2a-4265-92ac-dab15dbe095d" (UID: "b46f12a8-5d2a-4265-92ac-dab15dbe095d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:24.539168 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:24.539139 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b46f12a8-5d2a-4265-92ac-dab15dbe095d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46f12a8-5d2a-4265-92ac-dab15dbe095d-maas-api-service-ca\") pod \"b46f12a8-5d2a-4265-92ac-dab15dbe095d\" (UID: \"b46f12a8-5d2a-4265-92ac-dab15dbe095d\") " Nov 01 22:45:24.540534 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:24.540504 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djr46\" (UniqueName: \"kubernetes.io/projected/b46f12a8-5d2a-4265-92ac-dab15dbe095d-kube-api-access-djr46\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:24.540712 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:24.540550 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46f12a8-5d2a-4265-92ac-dab15dbe095d-maas-api-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:24.560198 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:24.560134 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-8gfsk" Nov 01 22:45:24.642047 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:24.641467 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f8111ee-a594-407a-92ee-48fa084f0fd1-kube-api-access-pwzst\" (UniqueName: \"kubernetes.io/projected/7f8111ee-a594-407a-92ee-48fa084f0fd1-kube-api-access-pwzst\") pod \"7f8111ee-a594-407a-92ee-48fa084f0fd1\" (UID: \"7f8111ee-a594-407a-92ee-48fa084f0fd1\") " Nov 01 22:45:24.642047 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:24.641613 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f8111ee-a594-407a-92ee-48fa084f0fd1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f8111ee-a594-407a-92ee-48fa084f0fd1-maas-api-service-ca\") pod \"7f8111ee-a594-407a-92ee-48fa084f0fd1\" (UID: \"7f8111ee-a594-407a-92ee-48fa084f0fd1\") " Nov 01 22:45:24.643481 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:24.643163 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f8111ee-a594-407a-92ee-48fa084f0fd1-maas-api-service-ca" pod "7f8111ee-a594-407a-92ee-48fa084f0fd1" (UID: "7f8111ee-a594-407a-92ee-48fa084f0fd1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:24.647305 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:24.647252 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f8111ee-a594-407a-92ee-48fa084f0fd1-kube-api-access-pwzst" pod "7f8111ee-a594-407a-92ee-48fa084f0fd1" (UID: "7f8111ee-a594-407a-92ee-48fa084f0fd1"). InnerVolumeSpecName "kube-api-access-pwzst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:24.739684 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:24.739600 5011 scope.go:122] "RemoveContainer" containerID="599f79892e5e6c2203d22d746bd61f44b0e6e41b135ba377083601cea303140e" Nov 01 22:45:24.745580 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:24.745474 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwzst\" (UniqueName: \"kubernetes.io/projected/7f8111ee-a594-407a-92ee-48fa084f0fd1-kube-api-access-pwzst\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:24.745580 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:24.745495 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f8111ee-a594-407a-92ee-48fa084f0fd1-maas-api-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:25.238789 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:25.238761 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-8gfsk" Nov 01 22:45:25.239244 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:25.238794 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892885-8gfsk" event={"ID":"7f8111ee-a594-407a-92ee-48fa084f0fd1","Type":"ContainerDied","Data":"c2d8d24cad97299fc2834f496e3cbbfb986cf0100cc09e43ffb18c23e6e2b37c"} Nov 01 22:45:25.239244 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:25.238869 5011 scope.go:122] "RemoveContainer" containerID="ea83ed8ee57eeaab03c0c43d1713326333c82fe1fe32f27d5f07ed696948d690" Nov 01 22:45:25.240652 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:25.240626 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p" Nov 01 22:45:25.240923 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:25.240621 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p" event={"ID":"b46f12a8-5d2a-4265-92ac-dab15dbe095d","Type":"ContainerDied","Data":"ccfcfdc66e0dd26e59a1b99ee0549b61e3db767ef61ceae8a51e38644d4ba1a7"} Nov 01 22:45:25.243631 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:25.243607 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892885-sqfmb" event={"ID":"600fad75-5eab-47e7-b929-a5cd12963bf0","Type":"ContainerStarted","Data":"b8da4dd300fe1ce13ce5fa49d1b056aba8ee73f3e5de7e76f12720757e40a7b2"} Nov 01 22:45:25.255883 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:25.255865 5011 scope.go:122] "RemoveContainer" containerID="dd73dd5ec88ec9cb103c59942dfc0a3ca26104fff42826c9e87cb29e559b0e9d" Nov 01 22:45:25.258544 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:25.258518 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75"] Nov 01 22:45:25.314707 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:25.314632 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892885-8gfsk"] Nov 01 22:45:25.326593 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:25.326557 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892885-8gfsk"] Nov 01 22:45:25.351210 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:25.351175 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p"] Nov 01 22:45:25.357625 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:25.357557 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a9f94e-29892885-mtx6p"] Nov 01 22:45:25.583244 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:25.583159 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75" Nov 01 22:45:25.681488 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:25.681456 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b6a3491-66c7-42a2-b43b-307bf20f52d1-kube-api-access-zbplh\" (UniqueName: \"kubernetes.io/projected/0b6a3491-66c7-42a2-b43b-307bf20f52d1-kube-api-access-zbplh\") pod \"0b6a3491-66c7-42a2-b43b-307bf20f52d1\" (UID: \"0b6a3491-66c7-42a2-b43b-307bf20f52d1\") " Nov 01 22:45:25.681631 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:25.681577 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b6a3491-66c7-42a2-b43b-307bf20f52d1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6a3491-66c7-42a2-b43b-307bf20f52d1-maas-api-service-ca\") pod \"0b6a3491-66c7-42a2-b43b-307bf20f52d1\" (UID: \"0b6a3491-66c7-42a2-b43b-307bf20f52d1\") " Nov 01 22:45:25.681964 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:25.681937 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b6a3491-66c7-42a2-b43b-307bf20f52d1-maas-api-service-ca" pod "0b6a3491-66c7-42a2-b43b-307bf20f52d1" (UID: "0b6a3491-66c7-42a2-b43b-307bf20f52d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:25.682285 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:25.682270 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6a3491-66c7-42a2-b43b-307bf20f52d1-maas-api-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:25.683772 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:25.683752 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6a3491-66c7-42a2-b43b-307bf20f52d1-kube-api-access-zbplh" pod "0b6a3491-66c7-42a2-b43b-307bf20f52d1" (UID: "0b6a3491-66c7-42a2-b43b-307bf20f52d1"). InnerVolumeSpecName "kube-api-access-zbplh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:25.750156 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:25.750119 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f8111ee-a594-407a-92ee-48fa084f0fd1" path="/var/lib/kubelet/pods/7f8111ee-a594-407a-92ee-48fa084f0fd1/volumes" Nov 01 22:45:25.750748 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:25.750733 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b46f12a8-5d2a-4265-92ac-dab15dbe095d" path="/var/lib/kubelet/pods/b46f12a8-5d2a-4265-92ac-dab15dbe095d/volumes" Nov 01 22:45:25.783583 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:25.783547 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbplh\" (UniqueName: \"kubernetes.io/projected/0b6a3491-66c7-42a2-b43b-307bf20f52d1-kube-api-access-zbplh\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:25.784175 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:45:25.784152 5011 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6a3491_66c7_42a2_b43b_307bf20f52d1.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:45:25.816925 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:25.816882 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892885-sqfmb"] Nov 01 22:45:26.345857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:26.343090 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75" event={"ID":"0b6a3491-66c7-42a2-b43b-307bf20f52d1","Type":"ContainerDied","Data":"d0ec851426990deaf5ab4003354f0020ab8ba00d38a6f6aeb780a6cad7b641cf"} Nov 01 22:45:26.345857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:26.343161 5011 scope.go:122] "RemoveContainer" containerID="8846ed175cd02f25d871fb67ed8955976d8137f450da33d84a119b80d47000dc" Nov 01 22:45:26.345857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:26.343303 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75" Nov 01 22:45:26.412974 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:26.411332 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75"] Nov 01 22:45:26.416500 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:26.416460 5011 generic.go:356] "Generic (PLEG): container finished" podID="600fad75-5eab-47e7-b929-a5cd12963bf0" containerID="b8da4dd300fe1ce13ce5fa49d1b056aba8ee73f3e5de7e76f12720757e40a7b2" exitCode=60 Nov 01 22:45:26.416668 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:26.416588 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892885-sqfmb" event={"ID":"600fad75-5eab-47e7-b929-a5cd12963bf0","Type":"ContainerDied","Data":"b8da4dd300fe1ce13ce5fa49d1b056aba8ee73f3e5de7e76f12720757e40a7b2"} Nov 01 22:45:26.456037 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:26.449822 5011 scope.go:122] "RemoveContainer" containerID="599f79892e5e6c2203d22d746bd61f44b0e6e41b135ba377083601cea303140e" Nov 01 22:45:26.510900 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:26.510785 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-df4b43-29892885-cqc75"] Nov 01 22:45:26.672989 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:26.672963 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892885-sqfmb" Nov 01 22:45:26.746357 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:26.746323 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/600fad75-5eab-47e7-b929-a5cd12963bf0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/600fad75-5eab-47e7-b929-a5cd12963bf0-maas-api-service-ca\") pod \"600fad75-5eab-47e7-b929-a5cd12963bf0\" (UID: \"600fad75-5eab-47e7-b929-a5cd12963bf0\") " Nov 01 22:45:26.746529 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:26.746508 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/600fad75-5eab-47e7-b929-a5cd12963bf0-kube-api-access-6l9qd\" (UniqueName: \"kubernetes.io/projected/600fad75-5eab-47e7-b929-a5cd12963bf0-kube-api-access-6l9qd\") pod \"600fad75-5eab-47e7-b929-a5cd12963bf0\" (UID: \"600fad75-5eab-47e7-b929-a5cd12963bf0\") " Nov 01 22:45:26.746772 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:26.746742 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/600fad75-5eab-47e7-b929-a5cd12963bf0-maas-api-service-ca" pod "600fad75-5eab-47e7-b929-a5cd12963bf0" (UID: "600fad75-5eab-47e7-b929-a5cd12963bf0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:26.747260 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:26.747234 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/600fad75-5eab-47e7-b929-a5cd12963bf0-maas-api-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:26.748889 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:26.748860 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/600fad75-5eab-47e7-b929-a5cd12963bf0-kube-api-access-6l9qd" pod "600fad75-5eab-47e7-b929-a5cd12963bf0" (UID: "600fad75-5eab-47e7-b929-a5cd12963bf0"). InnerVolumeSpecName "kube-api-access-6l9qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:26.849992 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:26.849956 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l9qd\" (UniqueName: \"kubernetes.io/projected/600fad75-5eab-47e7-b929-a5cd12963bf0-kube-api-access-6l9qd\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:27.429543 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:27.429345 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892885-sqfmb" event={"ID":"600fad75-5eab-47e7-b929-a5cd12963bf0","Type":"ContainerDied","Data":"e307fe89667745ccecebe333cd2d734251f31e4ad363d50bb56355793384b065"} Nov 01 22:45:27.429543 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:27.429404 5011 scope.go:122] "RemoveContainer" containerID="b8da4dd300fe1ce13ce5fa49d1b056aba8ee73f3e5de7e76f12720757e40a7b2" Nov 01 22:45:27.429543 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:27.429433 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892885-sqfmb" Nov 01 22:45:27.464048 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:27.464009 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892885-sqfmb"] Nov 01 22:45:27.468173 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:27.468145 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892885-sqfmb"] Nov 01 22:45:27.770907 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:27.768433 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b6a3491-66c7-42a2-b43b-307bf20f52d1" path="/var/lib/kubelet/pods/0b6a3491-66c7-42a2-b43b-307bf20f52d1/volumes" Nov 01 22:45:27.770907 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:27.768991 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="600fad75-5eab-47e7-b929-a5cd12963bf0" path="/var/lib/kubelet/pods/600fad75-5eab-47e7-b929-a5cd12963bf0/volumes" Nov 01 22:45:37.798141 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.798104 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2476f8-7564448745-xnlhh"] Nov 01 22:45:37.799010 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.798991 5011 cpu_manager.go:414] "containerMap: removing container" podUID="7f568e19-ff42-4413-a06f-34f9f9594077" containerName="maas-api" Nov 01 22:45:37.799093 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799012 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f568e19-ff42-4413-a06f-34f9f9594077" containerName="maas-api" Nov 01 22:45:37.799093 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799024 5011 cpu_manager.go:414] "containerMap: removing container" podUID="0b6a3491-66c7-42a2-b43b-307bf20f52d1" containerName="cleanup" Nov 01 22:45:37.799093 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799028 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b6a3491-66c7-42a2-b43b-307bf20f52d1" containerName="cleanup" Nov 01 22:45:37.799093 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799038 5011 cpu_manager.go:414] "containerMap: removing container" podUID="0b6a3491-66c7-42a2-b43b-307bf20f52d1" containerName="cleanup" Nov 01 22:45:37.799093 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799044 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b6a3491-66c7-42a2-b43b-307bf20f52d1" containerName="cleanup" Nov 01 22:45:37.799093 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799052 5011 cpu_manager.go:414] "containerMap: removing container" podUID="7f8111ee-a594-407a-92ee-48fa084f0fd1" containerName="cleanup" Nov 01 22:45:37.799093 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799056 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f8111ee-a594-407a-92ee-48fa084f0fd1" containerName="cleanup" Nov 01 22:45:37.799093 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799062 5011 cpu_manager.go:414] "containerMap: removing container" podUID="b46f12a8-5d2a-4265-92ac-dab15dbe095d" containerName="cleanup" Nov 01 22:45:37.799093 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799067 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b46f12a8-5d2a-4265-92ac-dab15dbe095d" containerName="cleanup" Nov 01 22:45:37.799093 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799079 5011 cpu_manager.go:414] "containerMap: removing container" podUID="0b6a3491-66c7-42a2-b43b-307bf20f52d1" containerName="cleanup" Nov 01 22:45:37.799093 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799083 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b6a3491-66c7-42a2-b43b-307bf20f52d1" containerName="cleanup" Nov 01 22:45:37.799093 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799089 5011 cpu_manager.go:414] "containerMap: removing container" podUID="7f8111ee-a594-407a-92ee-48fa084f0fd1" containerName="cleanup" Nov 01 22:45:37.799093 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799093 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f8111ee-a594-407a-92ee-48fa084f0fd1" containerName="cleanup" Nov 01 22:45:37.799093 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799100 5011 cpu_manager.go:414] "containerMap: removing container" podUID="b46f12a8-5d2a-4265-92ac-dab15dbe095d" containerName="cleanup" Nov 01 22:45:37.799551 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799104 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b46f12a8-5d2a-4265-92ac-dab15dbe095d" containerName="cleanup" Nov 01 22:45:37.799551 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799111 5011 cpu_manager.go:414] "containerMap: removing container" podUID="600fad75-5eab-47e7-b929-a5cd12963bf0" containerName="cleanup" Nov 01 22:45:37.799551 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799115 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="600fad75-5eab-47e7-b929-a5cd12963bf0" containerName="cleanup" Nov 01 22:45:37.799551 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799129 5011 cpu_manager.go:414] "containerMap: removing container" podUID="600fad75-5eab-47e7-b929-a5cd12963bf0" containerName="cleanup" Nov 01 22:45:37.799551 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799134 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="600fad75-5eab-47e7-b929-a5cd12963bf0" containerName="cleanup" Nov 01 22:45:37.799551 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799150 5011 cpu_manager.go:414] "containerMap: removing container" podUID="b46f12a8-5d2a-4265-92ac-dab15dbe095d" containerName="cleanup" Nov 01 22:45:37.799551 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799156 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b46f12a8-5d2a-4265-92ac-dab15dbe095d" containerName="cleanup" Nov 01 22:45:37.799551 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799166 5011 cpu_manager.go:414] "containerMap: removing container" podUID="600fad75-5eab-47e7-b929-a5cd12963bf0" containerName="cleanup" Nov 01 22:45:37.799551 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799172 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="600fad75-5eab-47e7-b929-a5cd12963bf0" containerName="cleanup" Nov 01 22:45:37.799551 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799198 5011 cpu_manager.go:414] "containerMap: removing container" podUID="7f8111ee-a594-407a-92ee-48fa084f0fd1" containerName="cleanup" Nov 01 22:45:37.799551 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799203 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f8111ee-a594-407a-92ee-48fa084f0fd1" containerName="cleanup" Nov 01 22:45:37.799551 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799302 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f568e19-ff42-4413-a06f-34f9f9594077" containerName="maas-api" Nov 01 22:45:37.799551 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799310 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b6a3491-66c7-42a2-b43b-307bf20f52d1" containerName="cleanup" Nov 01 22:45:37.799551 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799316 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b6a3491-66c7-42a2-b43b-307bf20f52d1" containerName="cleanup" Nov 01 22:45:37.799551 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799323 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f8111ee-a594-407a-92ee-48fa084f0fd1" containerName="cleanup" Nov 01 22:45:37.799551 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799329 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="b46f12a8-5d2a-4265-92ac-dab15dbe095d" containerName="cleanup" Nov 01 22:45:37.799551 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799334 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b6a3491-66c7-42a2-b43b-307bf20f52d1" containerName="cleanup" Nov 01 22:45:37.799551 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799339 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="b46f12a8-5d2a-4265-92ac-dab15dbe095d" containerName="cleanup" Nov 01 22:45:37.799551 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799344 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="600fad75-5eab-47e7-b929-a5cd12963bf0" containerName="cleanup" Nov 01 22:45:37.799551 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799351 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="600fad75-5eab-47e7-b929-a5cd12963bf0" containerName="cleanup" Nov 01 22:45:37.799551 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799358 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="600fad75-5eab-47e7-b929-a5cd12963bf0" containerName="cleanup" Nov 01 22:45:37.799551 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.799366 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f8111ee-a594-407a-92ee-48fa084f0fd1" containerName="cleanup" Nov 01 22:45:37.804787 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.804769 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2476f8-7564448745-xnlhh" Nov 01 22:45:37.813029 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.812991 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2476f8\"" Nov 01 22:45:37.819559 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.819536 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2476f8-7564448745-xnlhh"] Nov 01 22:45:37.855687 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.855659 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2476f8-dockercfg-l6584\"" Nov 01 22:45:37.860246 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.860222 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2476f8-7f7457c99-8f7vl"] Nov 01 22:45:37.861175 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.861156 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="b46f12a8-5d2a-4265-92ac-dab15dbe095d" containerName="cleanup" Nov 01 22:45:37.861274 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.861182 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f8111ee-a594-407a-92ee-48fa084f0fd1" containerName="cleanup" Nov 01 22:45:37.865112 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.865093 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2476f8-7f7457c99-8f7vl" Nov 01 22:45:38.018251 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.962360 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55rk4\" (UniqueName: \"kubernetes.io/projected/a5697b6f-7e9f-4705-8f34-698f2a5447ac-kube-api-access-55rk4\") pod \"payload-processing-e2e-api-a-2476f8-7564448745-xnlhh\" (UID: \"a5697b6f-7e9f-4705-8f34-698f2a5447ac\") " pod="openshift-ingress/payload-processing-e2e-api-a-2476f8-7564448745-xnlhh" Nov 01 22:45:38.018251 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.962477 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qvvr\" (UniqueName: \"kubernetes.io/projected/0e895bf2-3149-4322-a026-b2d1c72ca948-kube-api-access-7qvvr\") pod \"payload-pre-processing-e2e-api-a-2476f8-7f7457c99-8f7vl\" (UID: \"0e895bf2-3149-4322-a026-b2d1c72ca948\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2476f8-7f7457c99-8f7vl" Nov 01 22:45:38.018251 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.962539 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5697b6f-7e9f-4705-8f34-698f2a5447ac-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2476f8-7564448745-xnlhh\" (UID: \"a5697b6f-7e9f-4705-8f34-698f2a5447ac\") " pod="openshift-ingress/payload-processing-e2e-api-a-2476f8-7564448745-xnlhh" Nov 01 22:45:38.018251 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:37.962613 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e895bf2-3149-4322-a026-b2d1c72ca948-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2476f8-7f7457c99-8f7vl\" (UID: \"0e895bf2-3149-4322-a026-b2d1c72ca948\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2476f8-7f7457c99-8f7vl" Nov 01 22:45:38.021855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:38.020432 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2476f8-7f7457c99-8f7vl"] Nov 01 22:45:38.147004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:38.067663 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55rk4\" (UniqueName: \"kubernetes.io/projected/a5697b6f-7e9f-4705-8f34-698f2a5447ac-kube-api-access-55rk4\") pod \"payload-processing-e2e-api-a-2476f8-7564448745-xnlhh\" (UID: \"a5697b6f-7e9f-4705-8f34-698f2a5447ac\") " pod="openshift-ingress/payload-processing-e2e-api-a-2476f8-7564448745-xnlhh" Nov 01 22:45:38.147004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:38.067768 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qvvr\" (UniqueName: \"kubernetes.io/projected/0e895bf2-3149-4322-a026-b2d1c72ca948-kube-api-access-7qvvr\") pod \"payload-pre-processing-e2e-api-a-2476f8-7f7457c99-8f7vl\" (UID: \"0e895bf2-3149-4322-a026-b2d1c72ca948\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2476f8-7f7457c99-8f7vl" Nov 01 22:45:38.147004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:38.067905 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5697b6f-7e9f-4705-8f34-698f2a5447ac-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2476f8-7564448745-xnlhh\" (UID: \"a5697b6f-7e9f-4705-8f34-698f2a5447ac\") " pod="openshift-ingress/payload-processing-e2e-api-a-2476f8-7564448745-xnlhh" Nov 01 22:45:38.147004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:38.067973 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e895bf2-3149-4322-a026-b2d1c72ca948-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2476f8-7f7457c99-8f7vl\" (UID: \"0e895bf2-3149-4322-a026-b2d1c72ca948\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2476f8-7f7457c99-8f7vl" Nov 01 22:45:38.147004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:38.080321 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5697b6f-7e9f-4705-8f34-698f2a5447ac-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2476f8-7564448745-xnlhh\" (UID: \"a5697b6f-7e9f-4705-8f34-698f2a5447ac\") " pod="openshift-ingress/payload-processing-e2e-api-a-2476f8-7564448745-xnlhh" Nov 01 22:45:38.155201 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:38.155127 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e895bf2-3149-4322-a026-b2d1c72ca948-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2476f8-7f7457c99-8f7vl\" (UID: \"0e895bf2-3149-4322-a026-b2d1c72ca948\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2476f8-7f7457c99-8f7vl" Nov 01 22:45:38.245857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:38.241246 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55rk4\" (UniqueName: \"kubernetes.io/projected/a5697b6f-7e9f-4705-8f34-698f2a5447ac-kube-api-access-55rk4\") pod \"payload-processing-e2e-api-a-2476f8-7564448745-xnlhh\" (UID: \"a5697b6f-7e9f-4705-8f34-698f2a5447ac\") " pod="openshift-ingress/payload-processing-e2e-api-a-2476f8-7564448745-xnlhh" Nov 01 22:45:38.357358 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:38.357323 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qvvr\" (UniqueName: \"kubernetes.io/projected/0e895bf2-3149-4322-a026-b2d1c72ca948-kube-api-access-7qvvr\") pod \"payload-pre-processing-e2e-api-a-2476f8-7f7457c99-8f7vl\" (UID: \"0e895bf2-3149-4322-a026-b2d1c72ca948\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2476f8-7f7457c99-8f7vl" Nov 01 22:45:38.421032 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:38.421001 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2476f8-7564448745-xnlhh" Nov 01 22:45:38.447970 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:38.447932 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2476f8-7f7457c99-8f7vl" Nov 01 22:45:38.588515 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:38.588471 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-df4b43-886976559-l6xmq"] Nov 01 22:45:38.719940 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:45:38.719889 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5697b6f_7e9f_4705_8f34_698f2a5447ac.slice/crio-6c0669f74c6e8b4feee10f3dc3b50f71bd4b68c8d48467e704e0817811b7a29c WatchSource:0}: Error finding container 6c0669f74c6e8b4feee10f3dc3b50f71bd4b68c8d48467e704e0817811b7a29c: Status 404 returned error can't find the container with id 6c0669f74c6e8b4feee10f3dc3b50f71bd4b68c8d48467e704e0817811b7a29c Nov 01 22:45:38.720545 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:38.720435 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2476f8-7564448745-xnlhh"] Nov 01 22:45:38.760315 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:45:38.760287 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e895bf2_3149_4322_a026_b2d1c72ca948.slice/crio-f95551a633e042ee3f4d54a10e15e6b4a82594e3ff3a0463b55d98245e77a77c WatchSource:0}: Error finding container f95551a633e042ee3f4d54a10e15e6b4a82594e3ff3a0463b55d98245e77a77c: Status 404 returned error can't find the container with id f95551a633e042ee3f4d54a10e15e6b4a82594e3ff3a0463b55d98245e77a77c Nov 01 22:45:38.760939 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:38.760912 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2476f8-7f7457c99-8f7vl"] Nov 01 22:45:39.159917 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:39.157993 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7"] Nov 01 22:45:39.159917 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:39.158370 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" podUID="d81b97a5-fd37-4296-8c16-94f46421d4a3" containerName="istio-proxy" containerID="cri-o://559c98bafa33e8f3b1e8e3c6da6b70b260fff2325348be1dfb82919ee170e312" gracePeriod=30 Nov 01 22:45:39.599174 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:39.599139 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2476f8-7564448745-xnlhh" event={"ID":"a5697b6f-7e9f-4705-8f34-698f2a5447ac","Type":"ContainerStarted","Data":"44de70ac0415c04b052a40e222a28254c9c3fe2c23e7ac3abf67270b0c0f62a8"} Nov 01 22:45:39.599366 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:39.599194 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2476f8-7564448745-xnlhh" event={"ID":"a5697b6f-7e9f-4705-8f34-698f2a5447ac","Type":"ContainerStarted","Data":"6c0669f74c6e8b4feee10f3dc3b50f71bd4b68c8d48467e704e0817811b7a29c"} Nov 01 22:45:39.599366 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:39.599231 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-2476f8-7564448745-xnlhh" Nov 01 22:45:39.600891 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:39.600826 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2476f8-7f7457c99-8f7vl" event={"ID":"0e895bf2-3149-4322-a026-b2d1c72ca948","Type":"ContainerStarted","Data":"47181381fff81e95319c4eb3934eb5654400ed92050d3357c8fb911d58a2619e"} Nov 01 22:45:39.600996 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:39.600910 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2476f8-7f7457c99-8f7vl" Nov 01 22:45:39.600996 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:39.600931 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2476f8-7f7457c99-8f7vl" event={"ID":"0e895bf2-3149-4322-a026-b2d1c72ca948","Type":"ContainerStarted","Data":"f95551a633e042ee3f4d54a10e15e6b4a82594e3ff3a0463b55d98245e77a77c"} Nov 01 22:45:39.625609 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:39.625557 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-2476f8-7564448745-xnlhh" podStartSLOduration=2.625541363 podStartE2EDuration="2.625541363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:39.621717232 +0000 UTC m=+1308.026307865" watchObservedRunningTime="2026-11-01 22:45:39.625541363 +0000 UTC m=+1308.030131994" Nov 01 22:45:39.648543 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:39.648485 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2476f8-7f7457c99-8f7vl" podStartSLOduration=2.648465096 podStartE2EDuration="2.648465096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:39.643928307 +0000 UTC m=+1308.048518953" watchObservedRunningTime="2026-11-01 22:45:39.648465096 +0000 UTC m=+1308.053055730" Nov 01 22:45:43.627350 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:43.627298 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-df4b43-886976559-l6xmq" podUID="87414a3d-0620-434b-9736-32dd6ef0a8e1" containerName="payload-pre-processing" containerID="cri-o://24e02e783cfb5921a0d96e647dfa8659afefb74b55b33ba01c52e22db59d2082" gracePeriod=25 Nov 01 22:45:44.068204 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.068178 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-df4b43-886976559-l6xmq" Nov 01 22:45:44.107215 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.107185 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87414a3d-0620-434b-9736-32dd6ef0a8e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87414a3d-0620-434b-9736-32dd6ef0a8e1-plugins-config-volume\") pod \"87414a3d-0620-434b-9736-32dd6ef0a8e1\" (UID: \"87414a3d-0620-434b-9736-32dd6ef0a8e1\") " Nov 01 22:45:44.107402 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.107366 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87414a3d-0620-434b-9736-32dd6ef0a8e1-kube-api-access-zq6wm\" (UniqueName: \"kubernetes.io/projected/87414a3d-0620-434b-9736-32dd6ef0a8e1-kube-api-access-zq6wm\") pod \"87414a3d-0620-434b-9736-32dd6ef0a8e1\" (UID: \"87414a3d-0620-434b-9736-32dd6ef0a8e1\") " Nov 01 22:45:44.107623 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.107598 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87414a3d-0620-434b-9736-32dd6ef0a8e1-plugins-config-volume" pod "87414a3d-0620-434b-9736-32dd6ef0a8e1" (UID: "87414a3d-0620-434b-9736-32dd6ef0a8e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:44.108285 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.108250 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87414a3d-0620-434b-9736-32dd6ef0a8e1-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:44.110544 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.110504 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87414a3d-0620-434b-9736-32dd6ef0a8e1-kube-api-access-zq6wm" pod "87414a3d-0620-434b-9736-32dd6ef0a8e1" (UID: "87414a3d-0620-434b-9736-32dd6ef0a8e1"). InnerVolumeSpecName "kube-api-access-zq6wm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:44.209923 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.209864 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq6wm\" (UniqueName: \"kubernetes.io/projected/87414a3d-0620-434b-9736-32dd6ef0a8e1-kube-api-access-zq6wm\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:44.503654 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.503627 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:45:44.620753 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.620664 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-credential-socket\") pod \"d81b97a5-fd37-4296-8c16-94f46421d4a3\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " Nov 01 22:45:44.627858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.621107 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d81b97a5-fd37-4296-8c16-94f46421d4a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81b97a5-fd37-4296-8c16-94f46421d4a3-istio-podinfo\") pod \"d81b97a5-fd37-4296-8c16-94f46421d4a3\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " Nov 01 22:45:44.627858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.621179 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-workload-socket\") pod \"d81b97a5-fd37-4296-8c16-94f46421d4a3\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " Nov 01 22:45:44.627858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.621279 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d81b97a5-fd37-4296-8c16-94f46421d4a3-kube-api-access-b52t2\" (UniqueName: \"kubernetes.io/projected/d81b97a5-fd37-4296-8c16-94f46421d4a3-kube-api-access-b52t2\") pod \"d81b97a5-fd37-4296-8c16-94f46421d4a3\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " Nov 01 22:45:44.627858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.621311 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d81b97a5-fd37-4296-8c16-94f46421d4a3-istio-token\" (UniqueName: \"kubernetes.io/projected/d81b97a5-fd37-4296-8c16-94f46421d4a3-istio-token\") pod \"d81b97a5-fd37-4296-8c16-94f46421d4a3\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " Nov 01 22:45:44.627858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.621397 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-istio-envoy\") pod \"d81b97a5-fd37-4296-8c16-94f46421d4a3\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " Nov 01 22:45:44.627858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.621463 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d81b97a5-fd37-4296-8c16-94f46421d4a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81b97a5-fd37-4296-8c16-94f46421d4a3-istiod-ca-cert\") pod \"d81b97a5-fd37-4296-8c16-94f46421d4a3\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " Nov 01 22:45:44.627858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.621504 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-workload-certs\") pod \"d81b97a5-fd37-4296-8c16-94f46421d4a3\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " Nov 01 22:45:44.627858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.621546 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-istio-data\") pod \"d81b97a5-fd37-4296-8c16-94f46421d4a3\" (UID: \"d81b97a5-fd37-4296-8c16-94f46421d4a3\") " Nov 01 22:45:44.627858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.625782 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-credential-socket" pod "d81b97a5-fd37-4296-8c16-94f46421d4a3" (UID: "d81b97a5-fd37-4296-8c16-94f46421d4a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:44.636040 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.629453 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-workload-socket" pod "d81b97a5-fd37-4296-8c16-94f46421d4a3" (UID: "d81b97a5-fd37-4296-8c16-94f46421d4a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:44.636040 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.631036 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-workload-certs" pod "d81b97a5-fd37-4296-8c16-94f46421d4a3" (UID: "d81b97a5-fd37-4296-8c16-94f46421d4a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:44.636040 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.635068 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-istio-data" pod "d81b97a5-fd37-4296-8c16-94f46421d4a3" (UID: "d81b97a5-fd37-4296-8c16-94f46421d4a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:44.636040 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.635390 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-istio-envoy" pod "d81b97a5-fd37-4296-8c16-94f46421d4a3" (UID: "d81b97a5-fd37-4296-8c16-94f46421d4a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:44.636040 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.635910 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d81b97a5-fd37-4296-8c16-94f46421d4a3-istiod-ca-cert" pod "d81b97a5-fd37-4296-8c16-94f46421d4a3" (UID: "d81b97a5-fd37-4296-8c16-94f46421d4a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:44.639475 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.639113 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d81b97a5-fd37-4296-8c16-94f46421d4a3-kube-api-access-b52t2" pod "d81b97a5-fd37-4296-8c16-94f46421d4a3" (UID: "d81b97a5-fd37-4296-8c16-94f46421d4a3"). InnerVolumeSpecName "kube-api-access-b52t2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:44.642756 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.641969 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d81b97a5-fd37-4296-8c16-94f46421d4a3-istio-token" pod "d81b97a5-fd37-4296-8c16-94f46421d4a3" (UID: "d81b97a5-fd37-4296-8c16-94f46421d4a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:44.668857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.649642 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d81b97a5-fd37-4296-8c16-94f46421d4a3-istio-podinfo" pod "d81b97a5-fd37-4296-8c16-94f46421d4a3" (UID: "d81b97a5-fd37-4296-8c16-94f46421d4a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:45:44.731855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.725901 5011 generic.go:356] "Generic (PLEG): container finished" podID="87414a3d-0620-434b-9736-32dd6ef0a8e1" containerID="24e02e783cfb5921a0d96e647dfa8659afefb74b55b33ba01c52e22db59d2082" exitCode=0 Nov 01 22:45:44.731855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.726005 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-df4b43-886976559-l6xmq" event={"ID":"87414a3d-0620-434b-9736-32dd6ef0a8e1","Type":"ContainerDied","Data":"24e02e783cfb5921a0d96e647dfa8659afefb74b55b33ba01c52e22db59d2082"} Nov 01 22:45:44.731855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.726072 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-df4b43-886976559-l6xmq" event={"ID":"87414a3d-0620-434b-9736-32dd6ef0a8e1","Type":"ContainerDied","Data":"2d6945e35d2303767e18528ab48c8beacd35f6b6204d46b0ef6b6cc66537986e"} Nov 01 22:45:44.731855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.726093 5011 scope.go:122] "RemoveContainer" containerID="24e02e783cfb5921a0d96e647dfa8659afefb74b55b33ba01c52e22db59d2082" Nov 01 22:45:44.731855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.726194 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-df4b43-886976559-l6xmq" Nov 01 22:45:44.731855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.726544 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b52t2\" (UniqueName: \"kubernetes.io/projected/d81b97a5-fd37-4296-8c16-94f46421d4a3-kube-api-access-b52t2\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:44.731855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.726570 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81b97a5-fd37-4296-8c16-94f46421d4a3-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:44.731855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.726596 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:44.731855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.726614 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81b97a5-fd37-4296-8c16-94f46421d4a3-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:44.731855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.726628 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:44.731855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.726642 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:44.731855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.726657 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:44.731855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.726672 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81b97a5-fd37-4296-8c16-94f46421d4a3-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:44.731855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.726687 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81b97a5-fd37-4296-8c16-94f46421d4a3-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:44.755470 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.755413 5011 generic.go:356] "Generic (PLEG): container finished" podID="d81b97a5-fd37-4296-8c16-94f46421d4a3" containerID="559c98bafa33e8f3b1e8e3c6da6b70b260fff2325348be1dfb82919ee170e312" exitCode=0 Nov 01 22:45:44.755773 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.755689 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" event={"ID":"d81b97a5-fd37-4296-8c16-94f46421d4a3","Type":"ContainerDied","Data":"559c98bafa33e8f3b1e8e3c6da6b70b260fff2325348be1dfb82919ee170e312"} Nov 01 22:45:44.755773 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.755760 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" event={"ID":"d81b97a5-fd37-4296-8c16-94f46421d4a3","Type":"ContainerDied","Data":"ea6e44efc9c1b970fe985b2d377b6c5b480e58d3566bcdaf838cac24b4b89130"} Nov 01 22:45:44.755948 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.755900 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7" Nov 01 22:45:44.765973 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.765875 5011 scope.go:122] "RemoveContainer" containerID="24e02e783cfb5921a0d96e647dfa8659afefb74b55b33ba01c52e22db59d2082" Nov 01 22:45:44.766258 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:45:44.766238 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24e02e783cfb5921a0d96e647dfa8659afefb74b55b33ba01c52e22db59d2082\": container with ID starting with 24e02e783cfb5921a0d96e647dfa8659afefb74b55b33ba01c52e22db59d2082 not found: ID does not exist" containerID="24e02e783cfb5921a0d96e647dfa8659afefb74b55b33ba01c52e22db59d2082" Nov 01 22:45:44.766341 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.766267 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24e02e783cfb5921a0d96e647dfa8659afefb74b55b33ba01c52e22db59d2082"} err="failed to get container status \"24e02e783cfb5921a0d96e647dfa8659afefb74b55b33ba01c52e22db59d2082\": rpc error: code = NotFound desc = could not find container \"24e02e783cfb5921a0d96e647dfa8659afefb74b55b33ba01c52e22db59d2082\": container with ID starting with 24e02e783cfb5921a0d96e647dfa8659afefb74b55b33ba01c52e22db59d2082 not found: ID does not exist" Nov 01 22:45:44.766341 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.766283 5011 scope.go:122] "RemoveContainer" containerID="559c98bafa33e8f3b1e8e3c6da6b70b260fff2325348be1dfb82919ee170e312" Nov 01 22:45:44.778359 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.778338 5011 scope.go:122] "RemoveContainer" containerID="559c98bafa33e8f3b1e8e3c6da6b70b260fff2325348be1dfb82919ee170e312" Nov 01 22:45:44.778650 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:45:44.778630 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"559c98bafa33e8f3b1e8e3c6da6b70b260fff2325348be1dfb82919ee170e312\": container with ID starting with 559c98bafa33e8f3b1e8e3c6da6b70b260fff2325348be1dfb82919ee170e312 not found: ID does not exist" containerID="559c98bafa33e8f3b1e8e3c6da6b70b260fff2325348be1dfb82919ee170e312" Nov 01 22:45:44.778734 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.778655 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"559c98bafa33e8f3b1e8e3c6da6b70b260fff2325348be1dfb82919ee170e312"} err="failed to get container status \"559c98bafa33e8f3b1e8e3c6da6b70b260fff2325348be1dfb82919ee170e312\": rpc error: code = NotFound desc = could not find container \"559c98bafa33e8f3b1e8e3c6da6b70b260fff2325348be1dfb82919ee170e312\": container with ID starting with 559c98bafa33e8f3b1e8e3c6da6b70b260fff2325348be1dfb82919ee170e312 not found: ID does not exist" Nov 01 22:45:44.806450 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.806424 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-df4b43-886976559-l6xmq"] Nov 01 22:45:44.819184 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.819155 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-df4b43-886976559-l6xmq"] Nov 01 22:45:44.844541 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.844510 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7"] Nov 01 22:45:44.849396 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:44.849372 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-l6wp7"] Nov 01 22:45:45.748549 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:45.748520 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87414a3d-0620-434b-9736-32dd6ef0a8e1" path="/var/lib/kubelet/pods/87414a3d-0620-434b-9736-32dd6ef0a8e1/volumes" Nov 01 22:45:45.749095 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:45.749079 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d81b97a5-fd37-4296-8c16-94f46421d4a3" path="/var/lib/kubelet/pods/d81b97a5-fd37-4296-8c16-94f46421d4a3/volumes" Nov 01 22:45:46.963626 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:45:46.963588 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:45:48.680161 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:48.680125 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-c6cec6-5fb65c74cf-gzv7m"] Nov 01 22:45:48.680900 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:48.680879 5011 cpu_manager.go:414] "containerMap: removing container" podUID="87414a3d-0620-434b-9736-32dd6ef0a8e1" containerName="payload-pre-processing" Nov 01 22:45:48.680900 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:48.680895 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87414a3d-0620-434b-9736-32dd6ef0a8e1" containerName="payload-pre-processing" Nov 01 22:45:48.680996 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:48.680934 5011 cpu_manager.go:414] "containerMap: removing container" podUID="d81b97a5-fd37-4296-8c16-94f46421d4a3" containerName="istio-proxy" Nov 01 22:45:48.680996 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:48.680939 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d81b97a5-fd37-4296-8c16-94f46421d4a3" containerName="istio-proxy" Nov 01 22:45:48.681059 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:48.681049 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="87414a3d-0620-434b-9736-32dd6ef0a8e1" containerName="payload-pre-processing" Nov 01 22:45:48.681088 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:48.681064 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="d81b97a5-fd37-4296-8c16-94f46421d4a3" containerName="istio-proxy" Nov 01 22:45:48.686299 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:48.686279 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c6cec6-5fb65c74cf-gzv7m" Nov 01 22:45:48.702271 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:48.702248 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-c6cec6\"" Nov 01 22:45:48.702416 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:48.702400 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-c6cec6-dockercfg-lhn9h\"" Nov 01 22:45:48.704584 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:48.704560 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c6cec6-5dbc7cdf79-fv8g6"] Nov 01 22:45:48.709774 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:48.709755 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-c6cec6-5fb65c74cf-gzv7m"] Nov 01 22:45:48.709898 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:48.709807 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c6cec6-5dbc7cdf79-fv8g6" Nov 01 22:45:48.740478 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:48.740218 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c6cec6-5dbc7cdf79-fv8g6"] Nov 01 22:45:48.789857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:48.784538 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbpt4\" (UniqueName: \"kubernetes.io/projected/cc6e9b92-6af6-43b2-b88a-9565806323bc-kube-api-access-bbpt4\") pod \"payload-processing-e2e-shared-a-w1-c6cec6-5fb65c74cf-gzv7m\" (UID: \"cc6e9b92-6af6-43b2-b88a-9565806323bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c6cec6-5fb65c74cf-gzv7m" Nov 01 22:45:48.789857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:48.784615 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64lvs\" (UniqueName: \"kubernetes.io/projected/39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7-kube-api-access-64lvs\") pod \"payload-pre-processing-e2e-shared-a-w1-c6cec6-5dbc7cdf79-fv8g6\" (UID: \"39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c6cec6-5dbc7cdf79-fv8g6" Nov 01 22:45:48.789857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:48.784683 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc6e9b92-6af6-43b2-b88a-9565806323bc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-c6cec6-5fb65c74cf-gzv7m\" (UID: \"cc6e9b92-6af6-43b2-b88a-9565806323bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c6cec6-5fb65c74cf-gzv7m" Nov 01 22:45:48.789857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:48.784709 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-c6cec6-5dbc7cdf79-fv8g6\" (UID: \"39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c6cec6-5dbc7cdf79-fv8g6" Nov 01 22:45:48.995929 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:48.994470 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbpt4\" (UniqueName: \"kubernetes.io/projected/cc6e9b92-6af6-43b2-b88a-9565806323bc-kube-api-access-bbpt4\") pod \"payload-processing-e2e-shared-a-w1-c6cec6-5fb65c74cf-gzv7m\" (UID: \"cc6e9b92-6af6-43b2-b88a-9565806323bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c6cec6-5fb65c74cf-gzv7m" Nov 01 22:45:48.995929 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:48.994544 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64lvs\" (UniqueName: \"kubernetes.io/projected/39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7-kube-api-access-64lvs\") pod \"payload-pre-processing-e2e-shared-a-w1-c6cec6-5dbc7cdf79-fv8g6\" (UID: \"39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c6cec6-5dbc7cdf79-fv8g6" Nov 01 22:45:48.995929 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:48.994607 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc6e9b92-6af6-43b2-b88a-9565806323bc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-c6cec6-5fb65c74cf-gzv7m\" (UID: \"cc6e9b92-6af6-43b2-b88a-9565806323bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c6cec6-5fb65c74cf-gzv7m" Nov 01 22:45:48.995929 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:48.994633 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-c6cec6-5dbc7cdf79-fv8g6\" (UID: \"39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c6cec6-5dbc7cdf79-fv8g6" Nov 01 22:45:49.047855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:49.008126 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc6e9b92-6af6-43b2-b88a-9565806323bc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-c6cec6-5fb65c74cf-gzv7m\" (UID: \"cc6e9b92-6af6-43b2-b88a-9565806323bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c6cec6-5fb65c74cf-gzv7m" Nov 01 22:45:49.059965 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:49.012039 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-c6cec6-5dbc7cdf79-fv8g6\" (UID: \"39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c6cec6-5dbc7cdf79-fv8g6" Nov 01 22:45:49.197952 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:49.196505 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64lvs\" (UniqueName: \"kubernetes.io/projected/39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7-kube-api-access-64lvs\") pod \"payload-pre-processing-e2e-shared-a-w1-c6cec6-5dbc7cdf79-fv8g6\" (UID: \"39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c6cec6-5dbc7cdf79-fv8g6" Nov 01 22:45:49.208481 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:49.201972 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbpt4\" (UniqueName: \"kubernetes.io/projected/cc6e9b92-6af6-43b2-b88a-9565806323bc-kube-api-access-bbpt4\") pod \"payload-processing-e2e-shared-a-w1-c6cec6-5fb65c74cf-gzv7m\" (UID: \"cc6e9b92-6af6-43b2-b88a-9565806323bc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c6cec6-5fb65c74cf-gzv7m" Nov 01 22:45:49.302002 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:49.301917 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c6cec6-5fb65c74cf-gzv7m" Nov 01 22:45:49.365745 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:49.365689 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c6cec6-5dbc7cdf79-fv8g6" Nov 01 22:45:49.614190 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:45:49.614160 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6e9b92_6af6_43b2_b88a_9565806323bc.slice/crio-91eaaf07c9e842d54cf67dd747fdf5a067e7031f580f9b9f5f2a2cfe6a260e47 WatchSource:0}: Error finding container 91eaaf07c9e842d54cf67dd747fdf5a067e7031f580f9b9f5f2a2cfe6a260e47: Status 404 returned error can't find the container with id 91eaaf07c9e842d54cf67dd747fdf5a067e7031f580f9b9f5f2a2cfe6a260e47 Nov 01 22:45:49.614654 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:49.614632 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-c6cec6-5fb65c74cf-gzv7m"] Nov 01 22:45:49.682944 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:49.682914 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c6cec6-5dbc7cdf79-fv8g6"] Nov 01 22:45:49.683258 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:45:49.683127 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39cc4cd6_52b3_4ee8_8fa0_8018905ddcd7.slice/crio-3787ed537d661e018e342c4dfd9c1919cd67f454b44e87af2b8a80ac680090e1 WatchSource:0}: Error finding container 3787ed537d661e018e342c4dfd9c1919cd67f454b44e87af2b8a80ac680090e1: Status 404 returned error can't find the container with id 3787ed537d661e018e342c4dfd9c1919cd67f454b44e87af2b8a80ac680090e1 Nov 01 22:45:50.205664 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:50.205626 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c6cec6-5dbc7cdf79-fv8g6" event={"ID":"39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7","Type":"ContainerStarted","Data":"67b23cc879b660a9eb675764058dac8404c53bd19d65f6f36bbe6f35e62452f2"} Nov 01 22:45:50.205900 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:50.205717 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c6cec6-5dbc7cdf79-fv8g6" event={"ID":"39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7","Type":"ContainerStarted","Data":"3787ed537d661e018e342c4dfd9c1919cd67f454b44e87af2b8a80ac680090e1"} Nov 01 22:45:50.205900 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:50.205880 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c6cec6-5dbc7cdf79-fv8g6" Nov 01 22:45:50.207864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:50.207805 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c6cec6-5fb65c74cf-gzv7m" event={"ID":"cc6e9b92-6af6-43b2-b88a-9565806323bc","Type":"ContainerStarted","Data":"5c501b0add172d2f58632d62fe5d1f4d05b807075dea687e75afa49bfc800480"} Nov 01 22:45:50.207984 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:50.207889 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c6cec6-5fb65c74cf-gzv7m" Nov 01 22:45:50.207984 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:50.207909 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c6cec6-5fb65c74cf-gzv7m" event={"ID":"cc6e9b92-6af6-43b2-b88a-9565806323bc","Type":"ContainerStarted","Data":"91eaaf07c9e842d54cf67dd747fdf5a067e7031f580f9b9f5f2a2cfe6a260e47"} Nov 01 22:45:50.232657 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:50.232605 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c6cec6-5dbc7cdf79-fv8g6" podStartSLOduration=2.232588074 podStartE2EDuration="2.232588074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:50.229397719 +0000 UTC m=+1318.633988348" watchObservedRunningTime="2026-11-01 22:45:50.232588074 +0000 UTC m=+1318.637178707" Nov 01 22:45:50.260400 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:50.260341 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c6cec6-5fb65c74cf-gzv7m" podStartSLOduration=2.2603231790000002 podStartE2EDuration="2.260323179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:50.256903924 +0000 UTC m=+1318.661494555" watchObservedRunningTime="2026-11-01 22:45:50.260323179 +0000 UTC m=+1318.664913813" Nov 01 22:45:50.611488 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:50.611413 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-2476f8-7564448745-xnlhh" Nov 01 22:45:50.611655 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:50.611629 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2476f8-7f7457c99-8f7vl" Nov 01 22:45:57.544050 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:57.533037 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-4vmss"] Nov 01 22:45:57.875916 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:57.867986 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-4vmss" Nov 01 22:45:58.086645 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:58.086605 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-30578128-dockercfg-hgwzc\"" Nov 01 22:45:58.133936 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:58.115064 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-30578128\"" Nov 01 22:45:58.133936 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:58.115147 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 01 22:45:58.136790 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:58.136747 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:58.312797 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:58.183282 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htslt\" (UniqueName: \"kubernetes.io/projected/16f65edd-5dd3-4a6b-85e4-1fba8e61f207-kube-api-access-htslt\") pod \"payload-pre-processing-e2e-mt-30578128-796c56f57f-4vmss\" (UID: \"16f65edd-5dd3-4a6b-85e4-1fba8e61f207\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-4vmss" Nov 01 22:45:58.313563 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:58.313532 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16f65edd-5dd3-4a6b-85e4-1fba8e61f207-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-30578128-796c56f57f-4vmss\" (UID: \"16f65edd-5dd3-4a6b-85e4-1fba8e61f207\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-4vmss" Nov 01 22:45:58.314111 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:58.202496 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 01 22:45:58.324086 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:58.183573 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 22:45:58.324647 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:58.202649 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 01 22:45:58.349983 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:58.202686 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:58.349983 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:58.246764 5011 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:58.349983 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:58.291004 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 22:45:58.349983 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:58.202721 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:58.442729 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:58.417189 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16f65edd-5dd3-4a6b-85e4-1fba8e61f207-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-30578128-796c56f57f-4vmss\" (UID: \"16f65edd-5dd3-4a6b-85e4-1fba8e61f207\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-4vmss" Nov 01 22:45:58.546922 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:58.437008 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16f65edd-5dd3-4a6b-85e4-1fba8e61f207-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-30578128-796c56f57f-4vmss\" (UID: \"16f65edd-5dd3-4a6b-85e4-1fba8e61f207\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-4vmss" Nov 01 22:45:58.551182 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:58.496311 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-4vmss"] Nov 01 22:45:58.567101 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:58.556567 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htslt\" (UniqueName: \"kubernetes.io/projected/16f65edd-5dd3-4a6b-85e4-1fba8e61f207-kube-api-access-htslt\") pod \"payload-pre-processing-e2e-mt-30578128-796c56f57f-4vmss\" (UID: \"16f65edd-5dd3-4a6b-85e4-1fba8e61f207\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-4vmss" Nov 01 22:45:58.714973 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:58.714852 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htslt\" (UniqueName: \"kubernetes.io/projected/16f65edd-5dd3-4a6b-85e4-1fba8e61f207-kube-api-access-htslt\") pod \"payload-pre-processing-e2e-mt-30578128-796c56f57f-4vmss\" (UID: \"16f65edd-5dd3-4a6b-85e4-1fba8e61f207\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-4vmss" Nov 01 22:45:58.861999 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:58.861964 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-4vmss" Nov 01 22:45:59.185342 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:45:59.185315 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16f65edd_5dd3_4a6b_85e4_1fba8e61f207.slice/crio-7cdf4091dd28d356e54691fdd3b68c90469db16c967678680019dcd3df959779 WatchSource:0}: Error finding container 7cdf4091dd28d356e54691fdd3b68c90469db16c967678680019dcd3df959779: Status 404 returned error can't find the container with id 7cdf4091dd28d356e54691fdd3b68c90469db16c967678680019dcd3df959779 Nov 01 22:45:59.185473 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:59.185452 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-4vmss"] Nov 01 22:45:59.624272 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:59.624178 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-4vmss" event={"ID":"16f65edd-5dd3-4a6b-85e4-1fba8e61f207","Type":"ContainerStarted","Data":"2ac8069535645eb66d05f63237e6c40d5180ffb0a435d146a57f2042075a199e"} Nov 01 22:45:59.624817 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:59.624794 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-4vmss" event={"ID":"16f65edd-5dd3-4a6b-85e4-1fba8e61f207","Type":"ContainerStarted","Data":"7cdf4091dd28d356e54691fdd3b68c90469db16c967678680019dcd3df959779"} Nov 01 22:45:59.626027 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:59.626006 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-4vmss" Nov 01 22:45:59.667158 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:45:59.667108 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-4vmss" podStartSLOduration=2.667093012 podStartE2EDuration="2.667093012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:59.663460278 +0000 UTC m=+1328.068050911" watchObservedRunningTime="2026-11-01 22:45:59.667093012 +0000 UTC m=+1328.071683643" Nov 01 22:46:01.220514 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:01.220485 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c6cec6-5dbc7cdf79-fv8g6" Nov 01 22:46:01.220974 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:01.220568 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c6cec6-5fb65c74cf-gzv7m" Nov 01 22:46:09.272896 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:09.263892 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh"] Nov 01 22:46:09.272896 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:09.264123 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" podUID="324f88f5-9089-49ad-9a07-22abe2a0e98f" containerName="istio-proxy" containerID="cri-o://121915229adc2402e6e71be94af2d71d0587e69aae763c9d54e5088bbc4c6e84" gracePeriod=30 Nov 01 22:46:09.324213 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:09.281044 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg"] Nov 01 22:46:09.324213 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:09.281353 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" podUID="5a107568-bb23-4c0f-ad9e-cdfb206ce4bf" containerName="istio-proxy" containerID="cri-o://46c43d91908ed3a943a1ac875128656ebe9a2bb373a47f725e9af396f7b1c602" gracePeriod=30 Nov 01 22:46:09.324213 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:09.297287 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv"] Nov 01 22:46:09.324213 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:09.297556 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" podUID="5064b29e-328a-4a42-b129-1a6eba585a76" containerName="istio-proxy" containerID="cri-o://9c94aaab568401c9ad54dc83eacfe4bf61f2c6d2c5e4e4f857e21faf2131835f" gracePeriod=30 Nov 01 22:46:10.676071 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:10.676043 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-4vmss" Nov 01 22:46:12.587943 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:12.587899 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:46:12.588330 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:12.587969 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" podUID="5064b29e-328a-4a42-b129-1a6eba585a76" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:46:14.645606 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.645268 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:46:14.672878 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.663469 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-workload-certs\") pod \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " Nov 01 22:46:14.672878 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.663621 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istio-data\") pod \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " Nov 01 22:46:14.672878 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.663909 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-workload-socket\") pod \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " Nov 01 22:46:14.672878 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.663977 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istio-envoy\") pod \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " Nov 01 22:46:14.672878 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.664014 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-kube-api-access-v5fsb\" (UniqueName: \"kubernetes.io/projected/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-kube-api-access-v5fsb\") pod \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " Nov 01 22:46:14.672878 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.666272 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-workload-certs" pod "5a107568-bb23-4c0f-ad9e-cdfb206ce4bf" (UID: "5a107568-bb23-4c0f-ad9e-cdfb206ce4bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:14.672878 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.666682 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istio-data" pod "5a107568-bb23-4c0f-ad9e-cdfb206ce4bf" (UID: "5a107568-bb23-4c0f-ad9e-cdfb206ce4bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:14.672878 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.667066 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-workload-socket" pod "5a107568-bb23-4c0f-ad9e-cdfb206ce4bf" (UID: "5a107568-bb23-4c0f-ad9e-cdfb206ce4bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:14.672878 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.667402 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:14.672878 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.667420 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:14.672878 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.667443 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:14.672878 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.671777 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istio-envoy" pod "5a107568-bb23-4c0f-ad9e-cdfb206ce4bf" (UID: "5a107568-bb23-4c0f-ad9e-cdfb206ce4bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:14.673596 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.672971 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-kube-api-access-v5fsb" pod "5a107568-bb23-4c0f-ad9e-cdfb206ce4bf" (UID: "5a107568-bb23-4c0f-ad9e-cdfb206ce4bf"). InnerVolumeSpecName "kube-api-access-v5fsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:14.768477 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.768421 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istio-token\" (UniqueName: \"kubernetes.io/projected/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istio-token\") pod \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " Nov 01 22:46:14.768684 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.768532 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-credential-socket\") pod \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " Nov 01 22:46:14.768684 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.768676 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istio-podinfo\") pod \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " Nov 01 22:46:14.769028 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.768825 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istiod-ca-cert\") pod \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\" (UID: \"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf\") " Nov 01 22:46:14.769028 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.768963 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-credential-socket" pod "5a107568-bb23-4c0f-ad9e-cdfb206ce4bf" (UID: "5a107568-bb23-4c0f-ad9e-cdfb206ce4bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:14.769438 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.769397 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istiod-ca-cert" pod "5a107568-bb23-4c0f-ad9e-cdfb206ce4bf" (UID: "5a107568-bb23-4c0f-ad9e-cdfb206ce4bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:14.771302 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.771277 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istio-token" pod "5a107568-bb23-4c0f-ad9e-cdfb206ce4bf" (UID: "5a107568-bb23-4c0f-ad9e-cdfb206ce4bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:14.772146 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.771460 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istio-podinfo" pod "5a107568-bb23-4c0f-ad9e-cdfb206ce4bf" (UID: "5a107568-bb23-4c0f-ad9e-cdfb206ce4bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:46:14.772146 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.772139 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:14.772406 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.772162 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:14.772406 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.772177 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:14.772406 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.772192 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5fsb\" (UniqueName: \"kubernetes.io/projected/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-kube-api-access-v5fsb\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:14.772406 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.772205 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:14.772406 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.772218 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:14.887214 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.887185 5011 generic.go:356] "Generic (PLEG): container finished" podID="5a107568-bb23-4c0f-ad9e-cdfb206ce4bf" containerID="46c43d91908ed3a943a1ac875128656ebe9a2bb373a47f725e9af396f7b1c602" exitCode=0 Nov 01 22:46:14.887421 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.887398 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" event={"ID":"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf","Type":"ContainerDied","Data":"46c43d91908ed3a943a1ac875128656ebe9a2bb373a47f725e9af396f7b1c602"} Nov 01 22:46:14.887525 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.887495 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" event={"ID":"5a107568-bb23-4c0f-ad9e-cdfb206ce4bf","Type":"ContainerDied","Data":"8637bcb6db822b3961f2828588559b09ab1d68f8be4a78f7e47c5f072d29b444"} Nov 01 22:46:14.887588 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.887550 5011 scope.go:122] "RemoveContainer" containerID="46c43d91908ed3a943a1ac875128656ebe9a2bb373a47f725e9af396f7b1c602" Nov 01 22:46:14.887751 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.887732 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg" Nov 01 22:46:14.891279 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.891255 5011 generic.go:356] "Generic (PLEG): container finished" podID="324f88f5-9089-49ad-9a07-22abe2a0e98f" containerID="121915229adc2402e6e71be94af2d71d0587e69aae763c9d54e5088bbc4c6e84" exitCode=0 Nov 01 22:46:14.891391 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.891301 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" event={"ID":"324f88f5-9089-49ad-9a07-22abe2a0e98f","Type":"ContainerDied","Data":"121915229adc2402e6e71be94af2d71d0587e69aae763c9d54e5088bbc4c6e84"} Nov 01 22:46:14.893367 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.893343 5011 generic.go:356] "Generic (PLEG): container finished" podID="5064b29e-328a-4a42-b129-1a6eba585a76" containerID="9c94aaab568401c9ad54dc83eacfe4bf61f2c6d2c5e4e4f857e21faf2131835f" exitCode=0 Nov 01 22:46:14.893472 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.893411 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" event={"ID":"5064b29e-328a-4a42-b129-1a6eba585a76","Type":"ContainerDied","Data":"9c94aaab568401c9ad54dc83eacfe4bf61f2c6d2c5e4e4f857e21faf2131835f"} Nov 01 22:46:14.908523 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.908501 5011 scope.go:122] "RemoveContainer" containerID="46c43d91908ed3a943a1ac875128656ebe9a2bb373a47f725e9af396f7b1c602" Nov 01 22:46:14.908868 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:46:14.908809 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46c43d91908ed3a943a1ac875128656ebe9a2bb373a47f725e9af396f7b1c602\": container with ID starting with 46c43d91908ed3a943a1ac875128656ebe9a2bb373a47f725e9af396f7b1c602 not found: ID does not exist" containerID="46c43d91908ed3a943a1ac875128656ebe9a2bb373a47f725e9af396f7b1c602" Nov 01 22:46:14.908946 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.908878 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46c43d91908ed3a943a1ac875128656ebe9a2bb373a47f725e9af396f7b1c602"} err="failed to get container status \"46c43d91908ed3a943a1ac875128656ebe9a2bb373a47f725e9af396f7b1c602\": rpc error: code = NotFound desc = could not find container \"46c43d91908ed3a943a1ac875128656ebe9a2bb373a47f725e9af396f7b1c602\": container with ID starting with 46c43d91908ed3a943a1ac875128656ebe9a2bb373a47f725e9af396f7b1c602 not found: ID does not exist" Nov 01 22:46:14.941371 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.939814 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg"] Nov 01 22:46:14.951451 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:14.943733 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-s89pg"] Nov 01 22:46:15.144268 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.144234 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:46:15.179700 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.179670 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-workload-socket\") pod \"324f88f5-9089-49ad-9a07-22abe2a0e98f\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " Nov 01 22:46:15.179809 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.179728 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-workload-certs\") pod \"324f88f5-9089-49ad-9a07-22abe2a0e98f\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " Nov 01 22:46:15.180134 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.179963 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/324f88f5-9089-49ad-9a07-22abe2a0e98f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324f88f5-9089-49ad-9a07-22abe2a0e98f-istio-podinfo\") pod \"324f88f5-9089-49ad-9a07-22abe2a0e98f\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " Nov 01 22:46:15.180134 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.179959 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-workload-socket" pod "324f88f5-9089-49ad-9a07-22abe2a0e98f" (UID: "324f88f5-9089-49ad-9a07-22abe2a0e98f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:15.180134 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.180022 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-workload-certs" pod "324f88f5-9089-49ad-9a07-22abe2a0e98f" (UID: "324f88f5-9089-49ad-9a07-22abe2a0e98f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:15.180499 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.180483 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-istio-envoy\") pod \"324f88f5-9089-49ad-9a07-22abe2a0e98f\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " Nov 01 22:46:15.180604 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.180528 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-credential-socket\") pod \"324f88f5-9089-49ad-9a07-22abe2a0e98f\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " Nov 01 22:46:15.180604 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.180585 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/324f88f5-9089-49ad-9a07-22abe2a0e98f-kube-api-access-hfjfv\" (UniqueName: \"kubernetes.io/projected/324f88f5-9089-49ad-9a07-22abe2a0e98f-kube-api-access-hfjfv\") pod \"324f88f5-9089-49ad-9a07-22abe2a0e98f\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " Nov 01 22:46:15.180746 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.180728 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/324f88f5-9089-49ad-9a07-22abe2a0e98f-istio-token\" (UniqueName: \"kubernetes.io/projected/324f88f5-9089-49ad-9a07-22abe2a0e98f-istio-token\") pod \"324f88f5-9089-49ad-9a07-22abe2a0e98f\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " Nov 01 22:46:15.180866 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.180799 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-istio-data\") pod \"324f88f5-9089-49ad-9a07-22abe2a0e98f\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " Nov 01 22:46:15.180936 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.180906 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/324f88f5-9089-49ad-9a07-22abe2a0e98f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324f88f5-9089-49ad-9a07-22abe2a0e98f-istiod-ca-cert\") pod \"324f88f5-9089-49ad-9a07-22abe2a0e98f\" (UID: \"324f88f5-9089-49ad-9a07-22abe2a0e98f\") " Nov 01 22:46:15.181282 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.181254 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-istio-data" pod "324f88f5-9089-49ad-9a07-22abe2a0e98f" (UID: "324f88f5-9089-49ad-9a07-22abe2a0e98f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:15.181469 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.181446 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-credential-socket" pod "324f88f5-9089-49ad-9a07-22abe2a0e98f" (UID: "324f88f5-9089-49ad-9a07-22abe2a0e98f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:15.182123 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.182096 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/324f88f5-9089-49ad-9a07-22abe2a0e98f-istiod-ca-cert" pod "324f88f5-9089-49ad-9a07-22abe2a0e98f" (UID: "324f88f5-9089-49ad-9a07-22abe2a0e98f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:15.182789 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.182753 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/324f88f5-9089-49ad-9a07-22abe2a0e98f-istio-podinfo" pod "324f88f5-9089-49ad-9a07-22abe2a0e98f" (UID: "324f88f5-9089-49ad-9a07-22abe2a0e98f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:46:15.186799 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.186774 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:15.186932 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.186805 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324f88f5-9089-49ad-9a07-22abe2a0e98f-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:15.186932 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.186819 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:15.186932 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.186857 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:15.186932 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.186871 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324f88f5-9089-49ad-9a07-22abe2a0e98f-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:15.186932 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.186883 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:15.188163 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.188133 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/324f88f5-9089-49ad-9a07-22abe2a0e98f-kube-api-access-hfjfv" pod "324f88f5-9089-49ad-9a07-22abe2a0e98f" (UID: "324f88f5-9089-49ad-9a07-22abe2a0e98f"). InnerVolumeSpecName "kube-api-access-hfjfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:15.188283 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.188260 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-istio-envoy" pod "324f88f5-9089-49ad-9a07-22abe2a0e98f" (UID: "324f88f5-9089-49ad-9a07-22abe2a0e98f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:15.189765 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.189713 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/324f88f5-9089-49ad-9a07-22abe2a0e98f-istio-token" pod "324f88f5-9089-49ad-9a07-22abe2a0e98f" (UID: "324f88f5-9089-49ad-9a07-22abe2a0e98f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:15.194203 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.194181 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:46:15.288001 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.287961 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-istio-data\") pod \"5064b29e-328a-4a42-b129-1a6eba585a76\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " Nov 01 22:46:15.288202 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.288052 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-workload-socket\") pod \"5064b29e-328a-4a42-b129-1a6eba585a76\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " Nov 01 22:46:15.288202 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.288094 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5064b29e-328a-4a42-b129-1a6eba585a76-kube-api-access-nwwfc\" (UniqueName: \"kubernetes.io/projected/5064b29e-328a-4a42-b129-1a6eba585a76-kube-api-access-nwwfc\") pod \"5064b29e-328a-4a42-b129-1a6eba585a76\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " Nov 01 22:46:15.288202 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.288146 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-istio-envoy\") pod \"5064b29e-328a-4a42-b129-1a6eba585a76\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " Nov 01 22:46:15.288370 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.288210 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5064b29e-328a-4a42-b129-1a6eba585a76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5064b29e-328a-4a42-b129-1a6eba585a76-istiod-ca-cert\") pod \"5064b29e-328a-4a42-b129-1a6eba585a76\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " Nov 01 22:46:15.288370 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.288242 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5064b29e-328a-4a42-b129-1a6eba585a76-istio-token\" (UniqueName: \"kubernetes.io/projected/5064b29e-328a-4a42-b129-1a6eba585a76-istio-token\") pod \"5064b29e-328a-4a42-b129-1a6eba585a76\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " Nov 01 22:46:15.288370 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.288296 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-credential-socket\") pod \"5064b29e-328a-4a42-b129-1a6eba585a76\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " Nov 01 22:46:15.288370 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.288290 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-istio-data" pod "5064b29e-328a-4a42-b129-1a6eba585a76" (UID: "5064b29e-328a-4a42-b129-1a6eba585a76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:15.288370 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.288358 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-workload-socket" pod "5064b29e-328a-4a42-b129-1a6eba585a76" (UID: "5064b29e-328a-4a42-b129-1a6eba585a76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:15.288791 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.288729 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-credential-socket" pod "5064b29e-328a-4a42-b129-1a6eba585a76" (UID: "5064b29e-328a-4a42-b129-1a6eba585a76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:15.288942 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.288903 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5064b29e-328a-4a42-b129-1a6eba585a76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5064b29e-328a-4a42-b129-1a6eba585a76-istio-podinfo\") pod \"5064b29e-328a-4a42-b129-1a6eba585a76\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " Nov 01 22:46:15.289033 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.289007 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5064b29e-328a-4a42-b129-1a6eba585a76-istiod-ca-cert" pod "5064b29e-328a-4a42-b129-1a6eba585a76" (UID: "5064b29e-328a-4a42-b129-1a6eba585a76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:15.289106 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.289048 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-workload-certs\") pod \"5064b29e-328a-4a42-b129-1a6eba585a76\" (UID: \"5064b29e-328a-4a42-b129-1a6eba585a76\") " Nov 01 22:46:15.289644 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.289623 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-workload-certs" pod "5064b29e-328a-4a42-b129-1a6eba585a76" (UID: "5064b29e-328a-4a42-b129-1a6eba585a76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:15.290763 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.290188 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:15.290763 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.290212 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:15.290763 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.290227 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:15.290763 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.290240 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324f88f5-9089-49ad-9a07-22abe2a0e98f-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:15.290763 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.290254 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5064b29e-328a-4a42-b129-1a6eba585a76-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:15.290763 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.290268 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:15.290763 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.290281 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfjfv\" (UniqueName: \"kubernetes.io/projected/324f88f5-9089-49ad-9a07-22abe2a0e98f-kube-api-access-hfjfv\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:15.290763 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.290294 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324f88f5-9089-49ad-9a07-22abe2a0e98f-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:15.293566 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.293526 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5064b29e-328a-4a42-b129-1a6eba585a76-istio-podinfo" pod "5064b29e-328a-4a42-b129-1a6eba585a76" (UID: "5064b29e-328a-4a42-b129-1a6eba585a76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:46:15.298946 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.295277 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5064b29e-328a-4a42-b129-1a6eba585a76-istio-token" pod "5064b29e-328a-4a42-b129-1a6eba585a76" (UID: "5064b29e-328a-4a42-b129-1a6eba585a76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:15.302958 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.302922 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-istio-envoy" pod "5064b29e-328a-4a42-b129-1a6eba585a76" (UID: "5064b29e-328a-4a42-b129-1a6eba585a76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:15.305001 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.304979 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5064b29e-328a-4a42-b129-1a6eba585a76-kube-api-access-nwwfc" pod "5064b29e-328a-4a42-b129-1a6eba585a76" (UID: "5064b29e-328a-4a42-b129-1a6eba585a76"). InnerVolumeSpecName "kube-api-access-nwwfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:15.451858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.442077 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwwfc\" (UniqueName: \"kubernetes.io/projected/5064b29e-328a-4a42-b129-1a6eba585a76-kube-api-access-nwwfc\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:15.451858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.442118 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5064b29e-328a-4a42-b129-1a6eba585a76-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:15.451858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.442132 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5064b29e-328a-4a42-b129-1a6eba585a76-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:15.451858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.442144 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5064b29e-328a-4a42-b129-1a6eba585a76-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:15.755663 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.755579 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a107568-bb23-4c0f-ad9e-cdfb206ce4bf" path="/var/lib/kubelet/pods/5a107568-bb23-4c0f-ad9e-cdfb206ce4bf/volumes" Nov 01 22:46:15.947475 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.946065 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" event={"ID":"324f88f5-9089-49ad-9a07-22abe2a0e98f","Type":"ContainerDied","Data":"b089f9aa7c6260b405c64bd1afc25bb3e53a7bf414b58e92587e16b63a3df0d3"} Nov 01 22:46:15.947475 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.946154 5011 scope.go:122] "RemoveContainer" containerID="121915229adc2402e6e71be94af2d71d0587e69aae763c9d54e5088bbc4c6e84" Nov 01 22:46:15.947475 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.946278 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh" Nov 01 22:46:15.970090 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.970056 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" event={"ID":"5064b29e-328a-4a42-b129-1a6eba585a76","Type":"ContainerDied","Data":"34ace9d99df0b22d7207106bbe7ca8a13ffa163f3f7fd8b07daff3d47234d172"} Nov 01 22:46:15.970390 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.970373 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv" Nov 01 22:46:15.985021 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.984993 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh"] Nov 01 22:46:15.992274 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.992243 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-ch2jh"] Nov 01 22:46:15.993070 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:15.993053 5011 scope.go:122] "RemoveContainer" containerID="9c94aaab568401c9ad54dc83eacfe4bf61f2c6d2c5e4e4f857e21faf2131835f" Nov 01 22:46:16.010680 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:16.010599 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv"] Nov 01 22:46:16.015273 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:16.015249 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-2mnlv"] Nov 01 22:46:16.965738 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:46:16.965703 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:46:17.764658 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:17.764623 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="324f88f5-9089-49ad-9a07-22abe2a0e98f" path="/var/lib/kubelet/pods/324f88f5-9089-49ad-9a07-22abe2a0e98f/volumes" Nov 01 22:46:17.765801 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:17.765777 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5064b29e-328a-4a42-b129-1a6eba585a76" path="/var/lib/kubelet/pods/5064b29e-328a-4a42-b129-1a6eba585a76/volumes" Nov 01 22:46:20.864532 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:20.864498 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1799f8-fb9c976d7-6kds5"] Nov 01 22:46:20.865264 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:20.865248 5011 cpu_manager.go:414] "containerMap: removing container" podUID="5064b29e-328a-4a42-b129-1a6eba585a76" containerName="istio-proxy" Nov 01 22:46:20.865319 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:20.865265 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5064b29e-328a-4a42-b129-1a6eba585a76" containerName="istio-proxy" Nov 01 22:46:20.865319 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:20.865293 5011 cpu_manager.go:414] "containerMap: removing container" podUID="324f88f5-9089-49ad-9a07-22abe2a0e98f" containerName="istio-proxy" Nov 01 22:46:20.865319 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:20.865299 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="324f88f5-9089-49ad-9a07-22abe2a0e98f" containerName="istio-proxy" Nov 01 22:46:20.865319 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:20.865307 5011 cpu_manager.go:414] "containerMap: removing container" podUID="5a107568-bb23-4c0f-ad9e-cdfb206ce4bf" containerName="istio-proxy" Nov 01 22:46:20.865319 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:20.865313 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a107568-bb23-4c0f-ad9e-cdfb206ce4bf" containerName="istio-proxy" Nov 01 22:46:20.865473 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:20.865452 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="324f88f5-9089-49ad-9a07-22abe2a0e98f" containerName="istio-proxy" Nov 01 22:46:20.865473 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:20.865461 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a107568-bb23-4c0f-ad9e-cdfb206ce4bf" containerName="istio-proxy" Nov 01 22:46:20.865537 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:20.865473 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="5064b29e-328a-4a42-b129-1a6eba585a76" containerName="istio-proxy" Nov 01 22:46:20.871077 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:20.871054 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1799f8-fb9c976d7-6kds5" Nov 01 22:46:20.886528 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:20.886495 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1799f8-dockercfg-dnstb\"" Nov 01 22:46:20.886528 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:20.886500 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1799f8\"" Nov 01 22:46:20.890046 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:20.890007 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1799f8-fb9c976d7-6kds5"] Nov 01 22:46:20.947513 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:20.947475 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/389a14f5-f03c-494c-8f3c-7490c443ad6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1799f8-fb9c976d7-6kds5\" (UID: \"389a14f5-f03c-494c-8f3c-7490c443ad6c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1799f8-fb9c976d7-6kds5" Nov 01 22:46:20.947817 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:20.947796 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpbks\" (UniqueName: \"kubernetes.io/projected/389a14f5-f03c-494c-8f3c-7490c443ad6c-kube-api-access-rpbks\") pod \"payload-pre-processing-e2e-inf-a-1799f8-fb9c976d7-6kds5\" (UID: \"389a14f5-f03c-494c-8f3c-7490c443ad6c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1799f8-fb9c976d7-6kds5" Nov 01 22:46:21.065102 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:21.065056 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/389a14f5-f03c-494c-8f3c-7490c443ad6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1799f8-fb9c976d7-6kds5\" (UID: \"389a14f5-f03c-494c-8f3c-7490c443ad6c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1799f8-fb9c976d7-6kds5" Nov 01 22:46:21.065659 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:21.065631 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpbks\" (UniqueName: \"kubernetes.io/projected/389a14f5-f03c-494c-8f3c-7490c443ad6c-kube-api-access-rpbks\") pod \"payload-pre-processing-e2e-inf-a-1799f8-fb9c976d7-6kds5\" (UID: \"389a14f5-f03c-494c-8f3c-7490c443ad6c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1799f8-fb9c976d7-6kds5" Nov 01 22:46:21.067097 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:21.067071 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/389a14f5-f03c-494c-8f3c-7490c443ad6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1799f8-fb9c976d7-6kds5\" (UID: \"389a14f5-f03c-494c-8f3c-7490c443ad6c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1799f8-fb9c976d7-6kds5" Nov 01 22:46:21.103924 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:21.103886 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpbks\" (UniqueName: \"kubernetes.io/projected/389a14f5-f03c-494c-8f3c-7490c443ad6c-kube-api-access-rpbks\") pod \"payload-pre-processing-e2e-inf-a-1799f8-fb9c976d7-6kds5\" (UID: \"389a14f5-f03c-494c-8f3c-7490c443ad6c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1799f8-fb9c976d7-6kds5" Nov 01 22:46:21.189274 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:21.189233 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1799f8-fb9c976d7-6kds5" Nov 01 22:46:21.652011 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:21.651979 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1799f8-fb9c976d7-6kds5"] Nov 01 22:46:21.652324 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:46:21.652301 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod389a14f5_f03c_494c_8f3c_7490c443ad6c.slice/crio-04b0a4bab1a976c9a61ee6033b88d3f7aeb9041169fce173026e7f45491b116b WatchSource:0}: Error finding container 04b0a4bab1a976c9a61ee6033b88d3f7aeb9041169fce173026e7f45491b116b: Status 404 returned error can't find the container with id 04b0a4bab1a976c9a61ee6033b88d3f7aeb9041169fce173026e7f45491b116b Nov 01 22:46:22.363384 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:22.363345 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1799f8-fb9c976d7-6kds5" event={"ID":"389a14f5-f03c-494c-8f3c-7490c443ad6c","Type":"ContainerStarted","Data":"89ea92eccc0c30cfb5acb2afaad343781ab9cab2d635660cb5b650b03029b274"} Nov 01 22:46:22.363938 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:22.363408 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1799f8-fb9c976d7-6kds5" event={"ID":"389a14f5-f03c-494c-8f3c-7490c443ad6c","Type":"ContainerStarted","Data":"04b0a4bab1a976c9a61ee6033b88d3f7aeb9041169fce173026e7f45491b116b"} Nov 01 22:46:22.363938 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:22.363437 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1799f8-fb9c976d7-6kds5" Nov 01 22:46:22.400089 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:22.400020 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1799f8-fb9c976d7-6kds5" podStartSLOduration=2.400005797 podStartE2EDuration="2.400005797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:22.394239755 +0000 UTC m=+1350.798830388" watchObservedRunningTime="2026-11-01 22:46:22.400005797 +0000 UTC m=+1350.804596428" Nov 01 22:46:24.338930 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:24.338886 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97"] Nov 01 22:46:24.339405 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:24.339089 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" podUID="653a5287-e32f-41f0-a724-2e32dccff6a3" containerName="istio-proxy" containerID="cri-o://157e502c7cce3de39f0fab055448d23214780b6fcee931ab5ff9f12350779cf6" gracePeriod=30 Nov 01 22:46:24.343259 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:24.343231 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb"] Nov 01 22:46:24.343449 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:24.343430 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" podUID="a6d034ba-eb48-4ce8-b349-e2e816a123c1" containerName="istio-proxy" containerID="cri-o://638eec0f18190e29198806560afe5353c4525f0bed2548d2f6c726f4bf08d87b" gracePeriod=30 Nov 01 22:46:24.347186 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:24.347161 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths"] Nov 01 22:46:24.347347 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:24.347329 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" podUID="5ec80ece-abb9-45b5-a34e-45bbb2d90eb0" containerName="istio-proxy" containerID="cri-o://597fb365768a1a9a21f14c095594d7b749f20942526a54db7ef42a6e548f68f1" gracePeriod=30 Nov 01 22:46:27.099522 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:27.098907 5011 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:46:27.099522 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:27.098995 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" podUID="653a5287-e32f-41f0-a724-2e32dccff6a3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:46:29.482715 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:46:29.482667 5011 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653a5287_e32f_41f0_a724_2e32dccff6a3.slice/crio-157e502c7cce3de39f0fab055448d23214780b6fcee931ab5ff9f12350779cf6.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ec80ece_abb9_45b5_a34e_45bbb2d90eb0.slice/crio-conmon-597fb365768a1a9a21f14c095594d7b749f20942526a54db7ef42a6e548f68f1.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653a5287_e32f_41f0_a724_2e32dccff6a3.slice/crio-conmon-157e502c7cce3de39f0fab055448d23214780b6fcee931ab5ff9f12350779cf6.scope\": RecentStats: unable to find data in memory cache]" Nov 01 22:46:29.486948 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:29.486914 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2a5c3234-7c548f45c7-x6thk"] Nov 01 22:46:29.574156 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:29.574116 5011 generic.go:356] "Generic (PLEG): container finished" podID="a6d034ba-eb48-4ce8-b349-e2e816a123c1" containerID="638eec0f18190e29198806560afe5353c4525f0bed2548d2f6c726f4bf08d87b" exitCode=0 Nov 01 22:46:29.574482 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:29.574455 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" event={"ID":"a6d034ba-eb48-4ce8-b349-e2e816a123c1","Type":"ContainerDied","Data":"638eec0f18190e29198806560afe5353c4525f0bed2548d2f6c726f4bf08d87b"} Nov 01 22:46:29.578727 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:29.578698 5011 generic.go:356] "Generic (PLEG): container finished" podID="5ec80ece-abb9-45b5-a34e-45bbb2d90eb0" containerID="597fb365768a1a9a21f14c095594d7b749f20942526a54db7ef42a6e548f68f1" exitCode=0 Nov 01 22:46:29.579097 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:29.579075 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" event={"ID":"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0","Type":"ContainerDied","Data":"597fb365768a1a9a21f14c095594d7b749f20942526a54db7ef42a6e548f68f1"} Nov 01 22:46:29.582114 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:29.582091 5011 generic.go:356] "Generic (PLEG): container finished" podID="653a5287-e32f-41f0-a724-2e32dccff6a3" containerID="157e502c7cce3de39f0fab055448d23214780b6fcee931ab5ff9f12350779cf6" exitCode=0 Nov 01 22:46:29.582268 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:29.582250 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" event={"ID":"653a5287-e32f-41f0-a724-2e32dccff6a3","Type":"ContainerDied","Data":"157e502c7cce3de39f0fab055448d23214780b6fcee931ab5ff9f12350779cf6"} Nov 01 22:46:29.836424 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:29.836391 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:46:29.966038 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:29.965952 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6d034ba-eb48-4ce8-b349-e2e816a123c1-kube-api-access-hs8p9\" (UniqueName: \"kubernetes.io/projected/a6d034ba-eb48-4ce8-b349-e2e816a123c1-kube-api-access-hs8p9\") pod \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " Nov 01 22:46:29.982008 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:29.981977 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-workload-certs\") pod \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " Nov 01 22:46:30.085076 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.084954 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-credential-socket\") pod \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " Nov 01 22:46:30.085801 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.085690 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istiod-ca-cert\") pod \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " Nov 01 22:46:30.097877 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:29.988289 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-workload-certs" pod "a6d034ba-eb48-4ce8-b349-e2e816a123c1" (UID: "a6d034ba-eb48-4ce8-b349-e2e816a123c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:30.098141 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.097309 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istiod-ca-cert" pod "a6d034ba-eb48-4ce8-b349-e2e816a123c1" (UID: "a6d034ba-eb48-4ce8-b349-e2e816a123c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:30.098295 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.097635 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-credential-socket" pod "a6d034ba-eb48-4ce8-b349-e2e816a123c1" (UID: "a6d034ba-eb48-4ce8-b349-e2e816a123c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:30.098456 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.098440 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istio-envoy\") pod \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " Nov 01 22:46:30.105128 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.105090 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istio-data\") pod \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " Nov 01 22:46:30.129415 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.129384 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istio-podinfo\") pod \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " Nov 01 22:46:30.129855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.129817 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istio-token\" (UniqueName: \"kubernetes.io/projected/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istio-token\") pod \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " Nov 01 22:46:30.130089 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.130057 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-workload-socket\") pod \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\" (UID: \"a6d034ba-eb48-4ce8-b349-e2e816a123c1\") " Nov 01 22:46:30.148998 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.110002 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istio-data" pod "a6d034ba-eb48-4ce8-b349-e2e816a123c1" (UID: "a6d034ba-eb48-4ce8-b349-e2e816a123c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:30.168459 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.162395 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-workload-socket" pod "a6d034ba-eb48-4ce8-b349-e2e816a123c1" (UID: "a6d034ba-eb48-4ce8-b349-e2e816a123c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:30.168459 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.164066 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istio-envoy" pod "a6d034ba-eb48-4ce8-b349-e2e816a123c1" (UID: "a6d034ba-eb48-4ce8-b349-e2e816a123c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:30.168459 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.164132 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istio-token" pod "a6d034ba-eb48-4ce8-b349-e2e816a123c1" (UID: "a6d034ba-eb48-4ce8-b349-e2e816a123c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:30.171590 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.170724 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:30.171590 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.170750 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:30.171590 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.170761 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:30.171590 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.170769 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:30.171590 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.170778 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:30.171590 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.170786 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:30.171590 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.170807 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6d034ba-eb48-4ce8-b349-e2e816a123c1-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:30.172051 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.171995 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d034ba-eb48-4ce8-b349-e2e816a123c1-kube-api-access-hs8p9" pod "a6d034ba-eb48-4ce8-b349-e2e816a123c1" (UID: "a6d034ba-eb48-4ce8-b349-e2e816a123c1"). InnerVolumeSpecName "kube-api-access-hs8p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:30.172968 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.172944 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istio-podinfo" pod "a6d034ba-eb48-4ce8-b349-e2e816a123c1" (UID: "a6d034ba-eb48-4ce8-b349-e2e816a123c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:46:30.187216 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.187177 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:46:30.191219 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.191191 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:46:30.271649 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.271583 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istio-token\" (UniqueName: \"kubernetes.io/projected/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istio-token\") pod \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " Nov 01 22:46:30.271895 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.271876 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-workload-certs\") pod \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " Nov 01 22:46:30.272001 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.271991 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-credential-socket\") pod \"653a5287-e32f-41f0-a724-2e32dccff6a3\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " Nov 01 22:46:30.272081 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.272073 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istiod-ca-cert\") pod \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " Nov 01 22:46:30.272147 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.272137 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-workload-socket\") pod \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " Nov 01 22:46:30.272233 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.272218 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/653a5287-e32f-41f0-a724-2e32dccff6a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/653a5287-e32f-41f0-a724-2e32dccff6a3-istio-podinfo\") pod \"653a5287-e32f-41f0-a724-2e32dccff6a3\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " Nov 01 22:46:30.272374 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.272359 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-credential-socket\") pod \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " Nov 01 22:46:30.272513 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.272499 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istio-envoy\") pod \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " Nov 01 22:46:30.272703 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.272686 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-istio-envoy\") pod \"653a5287-e32f-41f0-a724-2e32dccff6a3\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " Nov 01 22:46:30.272863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.272849 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-istio-data\") pod \"653a5287-e32f-41f0-a724-2e32dccff6a3\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " Nov 01 22:46:30.273148 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.273132 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-workload-certs\") pod \"653a5287-e32f-41f0-a724-2e32dccff6a3\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " Nov 01 22:46:30.273290 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.273274 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/653a5287-e32f-41f0-a724-2e32dccff6a3-istio-token\" (UniqueName: \"kubernetes.io/projected/653a5287-e32f-41f0-a724-2e32dccff6a3-istio-token\") pod \"653a5287-e32f-41f0-a724-2e32dccff6a3\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " Nov 01 22:46:30.273484 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.273469 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/653a5287-e32f-41f0-a724-2e32dccff6a3-kube-api-access-rmhvb\" (UniqueName: \"kubernetes.io/projected/653a5287-e32f-41f0-a724-2e32dccff6a3-kube-api-access-rmhvb\") pod \"653a5287-e32f-41f0-a724-2e32dccff6a3\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " Nov 01 22:46:30.273628 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.273613 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-kube-api-access-nlw9t\" (UniqueName: \"kubernetes.io/projected/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-kube-api-access-nlw9t\") pod \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " Nov 01 22:46:30.273786 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.273773 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istio-data\") pod \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " Nov 01 22:46:30.273928 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.273914 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/653a5287-e32f-41f0-a724-2e32dccff6a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/653a5287-e32f-41f0-a724-2e32dccff6a3-istiod-ca-cert\") pod \"653a5287-e32f-41f0-a724-2e32dccff6a3\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " Nov 01 22:46:30.274075 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.274061 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istio-podinfo\") pod \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\" (UID: \"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0\") " Nov 01 22:46:30.274209 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.274196 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-workload-socket\") pod \"653a5287-e32f-41f0-a724-2e32dccff6a3\" (UID: \"653a5287-e32f-41f0-a724-2e32dccff6a3\") " Nov 01 22:46:30.275237 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.275195 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6d034ba-eb48-4ce8-b349-e2e816a123c1-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:30.275398 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.275377 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs8p9\" (UniqueName: \"kubernetes.io/projected/a6d034ba-eb48-4ce8-b349-e2e816a123c1-kube-api-access-hs8p9\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:30.275523 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.275495 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-workload-certs" pod "653a5287-e32f-41f0-a724-2e32dccff6a3" (UID: "653a5287-e32f-41f0-a724-2e32dccff6a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:30.275863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.275657 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-istio-data" pod "653a5287-e32f-41f0-a724-2e32dccff6a3" (UID: "653a5287-e32f-41f0-a724-2e32dccff6a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:30.275961 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.275900 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/653a5287-e32f-41f0-a724-2e32dccff6a3-istiod-ca-cert" pod "653a5287-e32f-41f0-a724-2e32dccff6a3" (UID: "653a5287-e32f-41f0-a724-2e32dccff6a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:30.276030 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.276006 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-workload-certs" pod "5ec80ece-abb9-45b5-a34e-45bbb2d90eb0" (UID: "5ec80ece-abb9-45b5-a34e-45bbb2d90eb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:30.276089 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.276028 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istiod-ca-cert" pod "5ec80ece-abb9-45b5-a34e-45bbb2d90eb0" (UID: "5ec80ece-abb9-45b5-a34e-45bbb2d90eb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:30.276089 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.276051 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istio-data" pod "5ec80ece-abb9-45b5-a34e-45bbb2d90eb0" (UID: "5ec80ece-abb9-45b5-a34e-45bbb2d90eb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:30.276346 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.276325 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-workload-socket" pod "653a5287-e32f-41f0-a724-2e32dccff6a3" (UID: "653a5287-e32f-41f0-a724-2e32dccff6a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:30.276471 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.276450 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-credential-socket" pod "653a5287-e32f-41f0-a724-2e32dccff6a3" (UID: "653a5287-e32f-41f0-a724-2e32dccff6a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:30.276630 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.276612 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-credential-socket" pod "5ec80ece-abb9-45b5-a34e-45bbb2d90eb0" (UID: "5ec80ece-abb9-45b5-a34e-45bbb2d90eb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:30.278087 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.278053 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-istio-envoy" pod "653a5287-e32f-41f0-a724-2e32dccff6a3" (UID: "653a5287-e32f-41f0-a724-2e32dccff6a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:30.278345 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.278321 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-workload-socket" pod "5ec80ece-abb9-45b5-a34e-45bbb2d90eb0" (UID: "5ec80ece-abb9-45b5-a34e-45bbb2d90eb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:30.279388 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.279359 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-kube-api-access-nlw9t" pod "5ec80ece-abb9-45b5-a34e-45bbb2d90eb0" (UID: "5ec80ece-abb9-45b5-a34e-45bbb2d90eb0"). InnerVolumeSpecName "kube-api-access-nlw9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:30.280189 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.280113 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istio-podinfo" pod "5ec80ece-abb9-45b5-a34e-45bbb2d90eb0" (UID: "5ec80ece-abb9-45b5-a34e-45bbb2d90eb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:46:30.280189 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.280153 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istio-envoy" pod "5ec80ece-abb9-45b5-a34e-45bbb2d90eb0" (UID: "5ec80ece-abb9-45b5-a34e-45bbb2d90eb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:30.280189 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.280179 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istio-token" pod "5ec80ece-abb9-45b5-a34e-45bbb2d90eb0" (UID: "5ec80ece-abb9-45b5-a34e-45bbb2d90eb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:30.280601 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.280279 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/653a5287-e32f-41f0-a724-2e32dccff6a3-kube-api-access-rmhvb" pod "653a5287-e32f-41f0-a724-2e32dccff6a3" (UID: "653a5287-e32f-41f0-a724-2e32dccff6a3"). InnerVolumeSpecName "kube-api-access-rmhvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:30.280906 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.280882 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/653a5287-e32f-41f0-a724-2e32dccff6a3-istio-token" pod "653a5287-e32f-41f0-a724-2e32dccff6a3" (UID: "653a5287-e32f-41f0-a724-2e32dccff6a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:30.282084 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.282060 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/653a5287-e32f-41f0-a724-2e32dccff6a3-istio-podinfo" pod "653a5287-e32f-41f0-a724-2e32dccff6a3" (UID: "653a5287-e32f-41f0-a724-2e32dccff6a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:46:30.377261 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.377225 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:30.377261 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.377259 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/653a5287-e32f-41f0-a724-2e32dccff6a3-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:30.377261 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.377270 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:30.377509 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.377280 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:30.377509 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.377288 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:30.377509 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.377297 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:30.377509 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.377305 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:30.377509 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.377314 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:30.377509 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.377323 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:30.377509 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.377331 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/653a5287-e32f-41f0-a724-2e32dccff6a3-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:30.377509 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.377339 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:30.377509 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.377348 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:30.377509 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.377356 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:30.377509 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.377364 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:30.377509 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.377372 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/653a5287-e32f-41f0-a724-2e32dccff6a3-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:30.377509 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.377379 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/653a5287-e32f-41f0-a724-2e32dccff6a3-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:30.377509 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.377388 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmhvb\" (UniqueName: \"kubernetes.io/projected/653a5287-e32f-41f0-a724-2e32dccff6a3-kube-api-access-rmhvb\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:30.377509 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.377397 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlw9t\" (UniqueName: \"kubernetes.io/projected/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0-kube-api-access-nlw9t\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:30.613947 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.611951 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" event={"ID":"a6d034ba-eb48-4ce8-b349-e2e816a123c1","Type":"ContainerDied","Data":"b951ff2d2e1718deb990260b6e34bc5e0e58bb44b90583b9923f9068c255aa61"} Nov 01 22:46:30.613947 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.612031 5011 scope.go:122] "RemoveContainer" containerID="638eec0f18190e29198806560afe5353c4525f0bed2548d2f6c726f4bf08d87b" Nov 01 22:46:30.613947 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.612204 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb" Nov 01 22:46:30.628473 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.621586 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" event={"ID":"5ec80ece-abb9-45b5-a34e-45bbb2d90eb0","Type":"ContainerDied","Data":"d86d77149130a2beade216808224552d4aa0ade74eff20df26f97364b9720814"} Nov 01 22:46:30.628473 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.622923 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths" Nov 01 22:46:30.652090 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.651980 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" Nov 01 22:46:30.652354 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.652322 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97" event={"ID":"653a5287-e32f-41f0-a724-2e32dccff6a3","Type":"ContainerDied","Data":"cf27aa8fd6b08cfe4ab71fc8ed2f4d3212e8a2aa540c71982f4f49bf23dbaf9e"} Nov 01 22:46:30.653762 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.653739 5011 scope.go:122] "RemoveContainer" containerID="597fb365768a1a9a21f14c095594d7b749f20942526a54db7ef42a6e548f68f1" Nov 01 22:46:30.671986 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.671947 5011 scope.go:122] "RemoveContainer" containerID="157e502c7cce3de39f0fab055448d23214780b6fcee931ab5ff9f12350779cf6" Nov 01 22:46:30.673134 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.673062 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb"] Nov 01 22:46:30.687574 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.685984 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-b45xb"] Nov 01 22:46:30.700765 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.700739 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths"] Nov 01 22:46:30.707649 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.707622 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-qmths"] Nov 01 22:46:30.720923 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.720895 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97"] Nov 01 22:46:30.727045 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:30.727023 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-x7v97"] Nov 01 22:46:31.750183 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:31.750097 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ec80ece-abb9-45b5-a34e-45bbb2d90eb0" path="/var/lib/kubelet/pods/5ec80ece-abb9-45b5-a34e-45bbb2d90eb0/volumes" Nov 01 22:46:31.750701 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:31.750684 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="653a5287-e32f-41f0-a724-2e32dccff6a3" path="/var/lib/kubelet/pods/653a5287-e32f-41f0-a724-2e32dccff6a3/volumes" Nov 01 22:46:31.751155 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:31.751141 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6d034ba-eb48-4ce8-b349-e2e816a123c1" path="/var/lib/kubelet/pods/a6d034ba-eb48-4ce8-b349-e2e816a123c1/volumes" Nov 01 22:46:33.298555 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.298518 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574"] Nov 01 22:46:33.299615 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.299593 5011 cpu_manager.go:414] "containerMap: removing container" podUID="653a5287-e32f-41f0-a724-2e32dccff6a3" containerName="istio-proxy" Nov 01 22:46:33.299615 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.299616 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="653a5287-e32f-41f0-a724-2e32dccff6a3" containerName="istio-proxy" Nov 01 22:46:33.299875 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.299640 5011 cpu_manager.go:414] "containerMap: removing container" podUID="5ec80ece-abb9-45b5-a34e-45bbb2d90eb0" containerName="istio-proxy" Nov 01 22:46:33.299875 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.299648 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ec80ece-abb9-45b5-a34e-45bbb2d90eb0" containerName="istio-proxy" Nov 01 22:46:33.299875 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.299695 5011 cpu_manager.go:414] "containerMap: removing container" podUID="a6d034ba-eb48-4ce8-b349-e2e816a123c1" containerName="istio-proxy" Nov 01 22:46:33.299875 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.299702 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6d034ba-eb48-4ce8-b349-e2e816a123c1" containerName="istio-proxy" Nov 01 22:46:33.300073 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.299917 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6d034ba-eb48-4ce8-b349-e2e816a123c1" containerName="istio-proxy" Nov 01 22:46:33.300073 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.299945 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="653a5287-e32f-41f0-a724-2e32dccff6a3" containerName="istio-proxy" Nov 01 22:46:33.300073 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.299973 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ec80ece-abb9-45b5-a34e-45bbb2d90eb0" containerName="istio-proxy" Nov 01 22:46:33.305490 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.305470 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:33.312709 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.312689 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-947846-openshift-default-dockercfg-qqfpg\"" Nov 01 22:46:33.324663 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.324634 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574"] Nov 01 22:46:33.455648 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.448138 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b054e19b-487f-4cde-9752-607550ccb6c7-istio-token\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:33.455648 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.448228 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b054e19b-487f-4cde-9752-607550ccb6c7-istio-podinfo\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:33.455648 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.448272 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-workload-certs\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:33.455648 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.448360 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-workload-socket\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:33.455648 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.448389 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-istio-envoy\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:33.455648 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.448519 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-istio-data\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:33.455648 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.448542 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b054e19b-487f-4cde-9752-607550ccb6c7-istiod-ca-cert\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:33.455648 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.448592 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v59nf\" (UniqueName: \"kubernetes.io/projected/b054e19b-487f-4cde-9752-607550ccb6c7-kube-api-access-v59nf\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:33.455648 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.448618 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-credential-socket\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:33.556250 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.556161 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-istio-data\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:33.556475 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.556456 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b054e19b-487f-4cde-9752-607550ccb6c7-istiod-ca-cert\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:33.556632 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.556617 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v59nf\" (UniqueName: \"kubernetes.io/projected/b054e19b-487f-4cde-9752-607550ccb6c7-kube-api-access-v59nf\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:33.556746 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.556732 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-credential-socket\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:33.557020 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.556998 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b054e19b-487f-4cde-9752-607550ccb6c7-istio-token\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:33.557199 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.557183 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b054e19b-487f-4cde-9752-607550ccb6c7-istio-podinfo\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:33.557320 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.557306 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-workload-certs\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:33.557490 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.557474 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-workload-socket\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:33.557603 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.557589 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-istio-envoy\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:33.558497 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.558477 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-credential-socket\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:33.558772 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.558753 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1799f8-fb9c976d7-6kds5" Nov 01 22:46:33.560371 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.560351 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-istio-data\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:33.572086 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.561067 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b054e19b-487f-4cde-9752-607550ccb6c7-istiod-ca-cert\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:33.575272 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.568292 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-workload-certs\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:33.575438 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.568536 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-workload-socket\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:33.575544 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.575205 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b054e19b-487f-4cde-9752-607550ccb6c7-istio-podinfo\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:33.627233 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.627202 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-istio-envoy\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:33.633500 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.633434 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b054e19b-487f-4cde-9752-607550ccb6c7-istio-token\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:33.640378 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.636483 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v59nf\" (UniqueName: \"kubernetes.io/projected/b054e19b-487f-4cde-9752-607550ccb6c7-kube-api-access-v59nf\") pod \"e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:33.921214 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:33.921186 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:34.243352 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:34.243246 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574"] Nov 01 22:46:34.247700 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:34.247662 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:46:34.247810 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:34.247742 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:46:34.247810 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:34.247789 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:46:34.487469 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:34.487417 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-2a5c3234-7c548f45c7-x6thk" podUID="53c8acf3-27ab-4ed5-87aa-1d64e8e015ab" containerName="payload-processing" containerID="cri-o://9bc5caf5778fe8281d263a28dda86c8a623cc5d2bf81ff6af5cf466d4dcaf2f7" gracePeriod=25 Nov 01 22:46:34.789935 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:34.789906 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2a5c3234-7c548f45c7-x6thk" Nov 01 22:46:34.849731 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:34.848872 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" event={"ID":"b054e19b-487f-4cde-9752-607550ccb6c7","Type":"ContainerStarted","Data":"da35b309d0236f9fdb10c5ed70695ef90d376636ba9b32715e055500e7fab362"} Nov 01 22:46:34.849731 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:34.848950 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" event={"ID":"b054e19b-487f-4cde-9752-607550ccb6c7","Type":"ContainerStarted","Data":"cfc2e01972998b9da625fe98e83ccb8a099047aebb2a1ccaa9d5091f855e1d9f"} Nov 01 22:46:34.850647 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:34.850619 5011 generic.go:356] "Generic (PLEG): container finished" podID="53c8acf3-27ab-4ed5-87aa-1d64e8e015ab" containerID="9bc5caf5778fe8281d263a28dda86c8a623cc5d2bf81ff6af5cf466d4dcaf2f7" exitCode=0 Nov 01 22:46:34.850812 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:34.850674 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2a5c3234-7c548f45c7-x6thk" event={"ID":"53c8acf3-27ab-4ed5-87aa-1d64e8e015ab","Type":"ContainerDied","Data":"9bc5caf5778fe8281d263a28dda86c8a623cc5d2bf81ff6af5cf466d4dcaf2f7"} Nov 01 22:46:34.850812 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:34.850717 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2a5c3234-7c548f45c7-x6thk" event={"ID":"53c8acf3-27ab-4ed5-87aa-1d64e8e015ab","Type":"ContainerDied","Data":"210ca37160937aa312ca994fdb4b8e295c0eca79a0e4256c8b5d2f48d48b31d8"} Nov 01 22:46:34.850812 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:34.850747 5011 scope.go:122] "RemoveContainer" containerID="9bc5caf5778fe8281d263a28dda86c8a623cc5d2bf81ff6af5cf466d4dcaf2f7" Nov 01 22:46:34.850992 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:34.850676 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2a5c3234-7c548f45c7-x6thk" Nov 01 22:46:34.885154 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:34.875804 5011 scope.go:122] "RemoveContainer" containerID="9bc5caf5778fe8281d263a28dda86c8a623cc5d2bf81ff6af5cf466d4dcaf2f7" Nov 01 22:46:34.885154 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:46:34.876209 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bc5caf5778fe8281d263a28dda86c8a623cc5d2bf81ff6af5cf466d4dcaf2f7\": container with ID starting with 9bc5caf5778fe8281d263a28dda86c8a623cc5d2bf81ff6af5cf466d4dcaf2f7 not found: ID does not exist" containerID="9bc5caf5778fe8281d263a28dda86c8a623cc5d2bf81ff6af5cf466d4dcaf2f7" Nov 01 22:46:34.885154 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:34.876244 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bc5caf5778fe8281d263a28dda86c8a623cc5d2bf81ff6af5cf466d4dcaf2f7"} err="failed to get container status \"9bc5caf5778fe8281d263a28dda86c8a623cc5d2bf81ff6af5cf466d4dcaf2f7\": rpc error: code = NotFound desc = could not find container \"9bc5caf5778fe8281d263a28dda86c8a623cc5d2bf81ff6af5cf466d4dcaf2f7\": container with ID starting with 9bc5caf5778fe8281d263a28dda86c8a623cc5d2bf81ff6af5cf466d4dcaf2f7 not found: ID does not exist" Nov 01 22:46:34.905940 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:34.904600 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" podStartSLOduration=1.9045698340000001 podStartE2EDuration="1.904569834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:34.899326175 +0000 UTC m=+1363.303916809" watchObservedRunningTime="2026-11-01 22:46:34.904569834 +0000 UTC m=+1363.309160474" Nov 01 22:46:34.987211 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:34.973815 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53c8acf3-27ab-4ed5-87aa-1d64e8e015ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53c8acf3-27ab-4ed5-87aa-1d64e8e015ab-plugins-config-volume\") pod \"53c8acf3-27ab-4ed5-87aa-1d64e8e015ab\" (UID: \"53c8acf3-27ab-4ed5-87aa-1d64e8e015ab\") " Nov 01 22:46:34.987211 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:34.974141 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53c8acf3-27ab-4ed5-87aa-1d64e8e015ab-kube-api-access-r8mw8\" (UniqueName: \"kubernetes.io/projected/53c8acf3-27ab-4ed5-87aa-1d64e8e015ab-kube-api-access-r8mw8\") pod \"53c8acf3-27ab-4ed5-87aa-1d64e8e015ab\" (UID: \"53c8acf3-27ab-4ed5-87aa-1d64e8e015ab\") " Nov 01 22:46:35.118963 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:35.097689 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:35.118963 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:35.117182 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53c8acf3-27ab-4ed5-87aa-1d64e8e015ab-plugins-config-volume" pod "53c8acf3-27ab-4ed5-87aa-1d64e8e015ab" (UID: "53c8acf3-27ab-4ed5-87aa-1d64e8e015ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:35.155730 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:35.155369 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:35.166532 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:35.166161 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53c8acf3-27ab-4ed5-87aa-1d64e8e015ab-kube-api-access-r8mw8" pod "53c8acf3-27ab-4ed5-87aa-1d64e8e015ab" (UID: "53c8acf3-27ab-4ed5-87aa-1d64e8e015ab"). InnerVolumeSpecName "kube-api-access-r8mw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:35.417787 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:35.405316 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8mw8\" (UniqueName: \"kubernetes.io/projected/53c8acf3-27ab-4ed5-87aa-1d64e8e015ab-kube-api-access-r8mw8\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:35.417787 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:35.405356 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53c8acf3-27ab-4ed5-87aa-1d64e8e015ab-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:35.622956 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:35.622909 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2a5c3234-7c548f45c7-x6thk"] Nov 01 22:46:35.694917 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:35.688469 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2a5c3234-7c548f45c7-x6thk"] Nov 01 22:46:35.792901 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:35.788235 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53c8acf3-27ab-4ed5-87aa-1d64e8e015ab" path="/var/lib/kubelet/pods/53c8acf3-27ab-4ed5-87aa-1d64e8e015ab/volumes" Nov 01 22:46:35.911916 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:35.909587 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:35.924879 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:35.924141 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:46:36.570512 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:36.570479 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:46:37.203569 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:37.203527 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm"] Nov 01 22:46:37.204768 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:37.204741 5011 cpu_manager.go:414] "containerMap: removing container" podUID="53c8acf3-27ab-4ed5-87aa-1d64e8e015ab" containerName="payload-processing" Nov 01 22:46:37.205022 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:37.205001 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53c8acf3-27ab-4ed5-87aa-1d64e8e015ab" containerName="payload-processing" Nov 01 22:46:37.205387 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:37.205370 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="53c8acf3-27ab-4ed5-87aa-1d64e8e015ab" containerName="payload-processing" Nov 01 22:46:37.211074 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:37.211053 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm" Nov 01 22:46:37.223906 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:37.223879 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm"] Nov 01 22:46:37.227691 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:37.225627 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1799f8-serving-cert\"" Nov 01 22:46:37.268479 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:37.268447 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6565ccc-c614-436a-8029-9e343d9ff296-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm\" (UID: \"b6565ccc-c614-436a-8029-9e343d9ff296\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm" Nov 01 22:46:37.268720 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:37.268676 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnzf4\" (UniqueName: \"kubernetes.io/projected/b6565ccc-c614-436a-8029-9e343d9ff296-kube-api-access-mnzf4\") pod \"maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm\" (UID: \"b6565ccc-c614-436a-8029-9e343d9ff296\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm" Nov 01 22:46:37.268807 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:37.268788 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6565ccc-c614-436a-8029-9e343d9ff296-maas-api-tls\") pod \"maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm\" (UID: \"b6565ccc-c614-436a-8029-9e343d9ff296\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm" Nov 01 22:46:37.268972 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:37.268954 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6565ccc-c614-436a-8029-9e343d9ff296-metrics-certs\") pod \"maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm\" (UID: \"b6565ccc-c614-436a-8029-9e343d9ff296\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm" Nov 01 22:46:37.371035 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:37.370998 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6565ccc-c614-436a-8029-9e343d9ff296-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm\" (UID: \"b6565ccc-c614-436a-8029-9e343d9ff296\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm" Nov 01 22:46:37.371230 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:37.371136 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnzf4\" (UniqueName: \"kubernetes.io/projected/b6565ccc-c614-436a-8029-9e343d9ff296-kube-api-access-mnzf4\") pod \"maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm\" (UID: \"b6565ccc-c614-436a-8029-9e343d9ff296\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm" Nov 01 22:46:37.371230 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:37.371193 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6565ccc-c614-436a-8029-9e343d9ff296-maas-api-tls\") pod \"maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm\" (UID: \"b6565ccc-c614-436a-8029-9e343d9ff296\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm" Nov 01 22:46:37.371351 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:37.371278 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6565ccc-c614-436a-8029-9e343d9ff296-metrics-certs\") pod \"maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm\" (UID: \"b6565ccc-c614-436a-8029-9e343d9ff296\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm" Nov 01 22:46:37.371729 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:37.371702 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6565ccc-c614-436a-8029-9e343d9ff296-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm\" (UID: \"b6565ccc-c614-436a-8029-9e343d9ff296\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm" Nov 01 22:46:37.374082 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:37.374061 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6565ccc-c614-436a-8029-9e343d9ff296-maas-api-tls\") pod \"maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm\" (UID: \"b6565ccc-c614-436a-8029-9e343d9ff296\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm" Nov 01 22:46:37.374170 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:37.374134 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6565ccc-c614-436a-8029-9e343d9ff296-metrics-certs\") pod \"maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm\" (UID: \"b6565ccc-c614-436a-8029-9e343d9ff296\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm" Nov 01 22:46:37.424884 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:37.422680 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnzf4\" (UniqueName: \"kubernetes.io/projected/b6565ccc-c614-436a-8029-9e343d9ff296-kube-api-access-mnzf4\") pod \"maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm\" (UID: \"b6565ccc-c614-436a-8029-9e343d9ff296\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm" Nov 01 22:46:37.550937 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:37.550550 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm" Nov 01 22:46:38.249559 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:38.248125 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm"] Nov 01 22:46:38.320931 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:38.318165 5011 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:46:38.948283 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:38.943229 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:46:39.151042 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:39.151004 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm" event={"ID":"b6565ccc-c614-436a-8029-9e343d9ff296","Type":"ContainerStarted","Data":"9589ed5334dc728aa879d75e9ef01500972131d3591ede5cb90eeee918018e04"} Nov 01 22:46:39.539590 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:39.539546 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-947846-578fb595bf-l7kpp"] Nov 01 22:46:39.547818 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:39.547785 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-947846-578fb595bf-l7kpp" Nov 01 22:46:39.558914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:39.558822 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-947846\"" Nov 01 22:46:39.561766 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:39.561740 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-947846-578fb595bf-l7kpp"] Nov 01 22:46:39.566539 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:39.566508 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-947846-7965f5db47-fr9tf"] Nov 01 22:46:39.572137 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:39.572114 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-947846-7965f5db47-fr9tf" Nov 01 22:46:39.590272 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:39.590245 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-947846-7965f5db47-fr9tf"] Nov 01 22:46:39.597337 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:39.597308 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-947846-dockercfg-lpmqk\"" Nov 01 22:46:39.741857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:39.736423 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bda6fa3-e106-47fb-91d8-1d150a0249a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-947846-7965f5db47-fr9tf\" (UID: \"6bda6fa3-e106-47fb-91d8-1d150a0249a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-947846-7965f5db47-fr9tf" Nov 01 22:46:39.741857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:39.737230 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn9gr\" (UniqueName: \"kubernetes.io/projected/6bda6fa3-e106-47fb-91d8-1d150a0249a8-kube-api-access-fn9gr\") pod \"payload-pre-processing-e2e-shared-b-w1-947846-7965f5db47-fr9tf\" (UID: \"6bda6fa3-e106-47fb-91d8-1d150a0249a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-947846-7965f5db47-fr9tf" Nov 01 22:46:39.741857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:39.737318 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcdf2\" (UniqueName: \"kubernetes.io/projected/9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729-kube-api-access-xcdf2\") pod \"payload-processing-e2e-shared-b-w1-947846-578fb595bf-l7kpp\" (UID: \"9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-947846-578fb595bf-l7kpp" Nov 01 22:46:39.741857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:39.737397 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-947846-578fb595bf-l7kpp\" (UID: \"9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-947846-578fb595bf-l7kpp" Nov 01 22:46:39.870916 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:39.870259 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn9gr\" (UniqueName: \"kubernetes.io/projected/6bda6fa3-e106-47fb-91d8-1d150a0249a8-kube-api-access-fn9gr\") pod \"payload-pre-processing-e2e-shared-b-w1-947846-7965f5db47-fr9tf\" (UID: \"6bda6fa3-e106-47fb-91d8-1d150a0249a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-947846-7965f5db47-fr9tf" Nov 01 22:46:39.870916 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:39.870355 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcdf2\" (UniqueName: \"kubernetes.io/projected/9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729-kube-api-access-xcdf2\") pod \"payload-processing-e2e-shared-b-w1-947846-578fb595bf-l7kpp\" (UID: \"9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-947846-578fb595bf-l7kpp" Nov 01 22:46:39.870916 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:39.870386 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-947846-578fb595bf-l7kpp\" (UID: \"9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-947846-578fb595bf-l7kpp" Nov 01 22:46:39.870916 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:39.870488 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bda6fa3-e106-47fb-91d8-1d150a0249a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-947846-7965f5db47-fr9tf\" (UID: \"6bda6fa3-e106-47fb-91d8-1d150a0249a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-947846-7965f5db47-fr9tf" Nov 01 22:46:39.893854 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:39.871689 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bda6fa3-e106-47fb-91d8-1d150a0249a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-947846-7965f5db47-fr9tf\" (UID: \"6bda6fa3-e106-47fb-91d8-1d150a0249a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-947846-7965f5db47-fr9tf" Nov 01 22:46:39.893854 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:39.872441 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-947846-578fb595bf-l7kpp\" (UID: \"9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-947846-578fb595bf-l7kpp" Nov 01 22:46:39.904866 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:39.903581 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn9gr\" (UniqueName: \"kubernetes.io/projected/6bda6fa3-e106-47fb-91d8-1d150a0249a8-kube-api-access-fn9gr\") pod \"payload-pre-processing-e2e-shared-b-w1-947846-7965f5db47-fr9tf\" (UID: \"6bda6fa3-e106-47fb-91d8-1d150a0249a8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-947846-7965f5db47-fr9tf" Nov 01 22:46:39.930923 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:39.923782 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcdf2\" (UniqueName: \"kubernetes.io/projected/9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729-kube-api-access-xcdf2\") pod \"payload-processing-e2e-shared-b-w1-947846-578fb595bf-l7kpp\" (UID: \"9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-947846-578fb595bf-l7kpp" Nov 01 22:46:40.162172 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:40.162086 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm" event={"ID":"b6565ccc-c614-436a-8029-9e343d9ff296","Type":"ContainerStarted","Data":"efe77cb92b575a47fc26c58554c54fdac89e4f6d7914292eccf01d1f3012d04d"} Nov 01 22:46:40.162372 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:40.162218 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm" Nov 01 22:46:40.167922 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:40.167894 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-947846-578fb595bf-l7kpp" Nov 01 22:46:40.191679 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:40.191619 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm" podStartSLOduration=2.46506822 podStartE2EDuration="3.191600687s" totalImagesPullingTime="726.532467ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:40.185247995 +0000 UTC m=+1368.589838650" watchObservedRunningTime="2026-11-01 22:46:40.191600687 +0000 UTC m=+1368.596191321" Nov 01 22:46:40.193684 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:40.193660 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-947846-7965f5db47-fr9tf" Nov 01 22:46:40.724223 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:40.721994 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-947846-578fb595bf-l7kpp"] Nov 01 22:46:40.724223 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:46:40.723585 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b0e4bcc_a8fd_4c57_8ae8_4b2c730dc729.slice/crio-d963d7caac1cc1a3697e5bbd74b4a5bbe74d22a6f2c4adb710f8d511631f0030 WatchSource:0}: Error finding container d963d7caac1cc1a3697e5bbd74b4a5bbe74d22a6f2c4adb710f8d511631f0030: Status 404 returned error can't find the container with id d963d7caac1cc1a3697e5bbd74b4a5bbe74d22a6f2c4adb710f8d511631f0030 Nov 01 22:46:40.791463 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:40.791172 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-947846-7965f5db47-fr9tf"] Nov 01 22:46:41.171493 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:41.171456 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-947846-578fb595bf-l7kpp" event={"ID":"9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729","Type":"ContainerStarted","Data":"12a03fdb3b9d6713424de68e36f661cd49058a089f11a547ad861208f4d452d2"} Nov 01 22:46:41.171691 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:41.171520 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-947846-578fb595bf-l7kpp" event={"ID":"9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729","Type":"ContainerStarted","Data":"d963d7caac1cc1a3697e5bbd74b4a5bbe74d22a6f2c4adb710f8d511631f0030"} Nov 01 22:46:41.171691 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:41.171548 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-947846-578fb595bf-l7kpp" Nov 01 22:46:41.173301 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:41.173277 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-947846-7965f5db47-fr9tf" event={"ID":"6bda6fa3-e106-47fb-91d8-1d150a0249a8","Type":"ContainerStarted","Data":"971e5751bf228ceff3e5a624c1e3662a33416bb29eb881e5fea22504ab0462c2"} Nov 01 22:46:41.173413 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:41.173321 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-947846-7965f5db47-fr9tf" event={"ID":"6bda6fa3-e106-47fb-91d8-1d150a0249a8","Type":"ContainerStarted","Data":"af4bcb661f2fae43d3eb1e64228b6105fd64bfb8b23b11f715adac9bff7af32c"} Nov 01 22:46:41.192237 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:41.192190 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-947846-578fb595bf-l7kpp" podStartSLOduration=2.192178151 podStartE2EDuration="2.192178151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:41.189479947 +0000 UTC m=+1369.594070581" watchObservedRunningTime="2026-11-01 22:46:41.192178151 +0000 UTC m=+1369.596768782" Nov 01 22:46:41.211552 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:41.211508 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-947846-7965f5db47-fr9tf" podStartSLOduration=2.211494512 podStartE2EDuration="2.211494512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:41.209631849 +0000 UTC m=+1369.614222482" watchObservedRunningTime="2026-11-01 22:46:41.211494512 +0000 UTC m=+1369.616085161" Nov 01 22:46:41.630888 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:41.630777 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c0d076-9775f9644-dn545"] Nov 01 22:46:41.636875 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:41.636810 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c0d076-9775f9644-dn545" Nov 01 22:46:41.648664 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:41.648621 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-c0d076-serving-cert\"" Nov 01 22:46:41.651994 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:41.651177 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c0d076-9775f9644-dn545"] Nov 01 22:46:41.735572 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:41.735534 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb5m8\" (UniqueName: \"kubernetes.io/projected/51471588-b877-4ffe-ba04-0e80882dfcd6-kube-api-access-jb5m8\") pod \"maas-api-e2e-api-b-c0d076-9775f9644-dn545\" (UID: \"51471588-b877-4ffe-ba04-0e80882dfcd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c0d076-9775f9644-dn545" Nov 01 22:46:41.735572 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:41.735575 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51471588-b877-4ffe-ba04-0e80882dfcd6-maas-api-tls\") pod \"maas-api-e2e-api-b-c0d076-9775f9644-dn545\" (UID: \"51471588-b877-4ffe-ba04-0e80882dfcd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c0d076-9775f9644-dn545" Nov 01 22:46:41.736032 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:41.735602 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51471588-b877-4ffe-ba04-0e80882dfcd6-openshift-service-ca\") pod \"maas-api-e2e-api-b-c0d076-9775f9644-dn545\" (UID: \"51471588-b877-4ffe-ba04-0e80882dfcd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c0d076-9775f9644-dn545" Nov 01 22:46:41.736032 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:41.735765 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51471588-b877-4ffe-ba04-0e80882dfcd6-metrics-certs\") pod \"maas-api-e2e-api-b-c0d076-9775f9644-dn545\" (UID: \"51471588-b877-4ffe-ba04-0e80882dfcd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c0d076-9775f9644-dn545" Nov 01 22:46:41.847856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:41.842355 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb5m8\" (UniqueName: \"kubernetes.io/projected/51471588-b877-4ffe-ba04-0e80882dfcd6-kube-api-access-jb5m8\") pod \"maas-api-e2e-api-b-c0d076-9775f9644-dn545\" (UID: \"51471588-b877-4ffe-ba04-0e80882dfcd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c0d076-9775f9644-dn545" Nov 01 22:46:41.847856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:41.842409 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51471588-b877-4ffe-ba04-0e80882dfcd6-maas-api-tls\") pod \"maas-api-e2e-api-b-c0d076-9775f9644-dn545\" (UID: \"51471588-b877-4ffe-ba04-0e80882dfcd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c0d076-9775f9644-dn545" Nov 01 22:46:41.847856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:41.842441 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51471588-b877-4ffe-ba04-0e80882dfcd6-openshift-service-ca\") pod \"maas-api-e2e-api-b-c0d076-9775f9644-dn545\" (UID: \"51471588-b877-4ffe-ba04-0e80882dfcd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c0d076-9775f9644-dn545" Nov 01 22:46:41.847856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:41.842630 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51471588-b877-4ffe-ba04-0e80882dfcd6-metrics-certs\") pod \"maas-api-e2e-api-b-c0d076-9775f9644-dn545\" (UID: \"51471588-b877-4ffe-ba04-0e80882dfcd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c0d076-9775f9644-dn545" Nov 01 22:46:41.847856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:41.845058 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51471588-b877-4ffe-ba04-0e80882dfcd6-openshift-service-ca\") pod \"maas-api-e2e-api-b-c0d076-9775f9644-dn545\" (UID: \"51471588-b877-4ffe-ba04-0e80882dfcd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c0d076-9775f9644-dn545" Nov 01 22:46:41.892819 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:41.892645 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51471588-b877-4ffe-ba04-0e80882dfcd6-maas-api-tls\") pod \"maas-api-e2e-api-b-c0d076-9775f9644-dn545\" (UID: \"51471588-b877-4ffe-ba04-0e80882dfcd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c0d076-9775f9644-dn545" Nov 01 22:46:41.905941 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:41.893495 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51471588-b877-4ffe-ba04-0e80882dfcd6-metrics-certs\") pod \"maas-api-e2e-api-b-c0d076-9775f9644-dn545\" (UID: \"51471588-b877-4ffe-ba04-0e80882dfcd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c0d076-9775f9644-dn545" Nov 01 22:46:41.905941 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:41.899552 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb5m8\" (UniqueName: \"kubernetes.io/projected/51471588-b877-4ffe-ba04-0e80882dfcd6-kube-api-access-jb5m8\") pod \"maas-api-e2e-api-b-c0d076-9775f9644-dn545\" (UID: \"51471588-b877-4ffe-ba04-0e80882dfcd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c0d076-9775f9644-dn545" Nov 01 22:46:41.989505 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:41.989471 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c0d076-9775f9644-dn545" Nov 01 22:46:42.184634 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:42.184584 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-947846-7965f5db47-fr9tf" Nov 01 22:46:42.386869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:42.359604 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c0d076-9775f9644-dn545"] Nov 01 22:46:43.195176 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:43.195097 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c0d076-9775f9644-dn545" event={"ID":"51471588-b877-4ffe-ba04-0e80882dfcd6","Type":"ContainerStarted","Data":"68e5febefcf341b606a34caf78faffb45e743a40bfa496018fe87a24f4f41092"} Nov 01 22:46:43.195176 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:43.195151 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c0d076-9775f9644-dn545" event={"ID":"51471588-b877-4ffe-ba04-0e80882dfcd6","Type":"ContainerStarted","Data":"cbdafd767840c7ef5c9606a553b8b37f88e924aa1771229981ce342a0b60c7d3"} Nov 01 22:46:43.195602 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:43.195187 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c0d076-9775f9644-dn545" Nov 01 22:46:43.223351 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:43.223300 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c0d076-9775f9644-dn545" podStartSLOduration=1.6955721270000002 podStartE2EDuration="2.223285648s" totalImagesPullingTime="527.713521ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:43.219549484 +0000 UTC m=+1371.624140128" watchObservedRunningTime="2026-11-01 22:46:43.223285648 +0000 UTC m=+1371.627876279" Nov 01 22:46:43.800977 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:43.800943 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r"] Nov 01 22:46:43.801284 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:43.801254 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" podUID="58c1620e-7736-4fe3-bfde-8024cfb8d2e5" containerName="istio-proxy" containerID="cri-o://52f591d9f3e3a4645a5195318b13b1b4272e64b6b0f5785d56cc7e5fc93e1747" gracePeriod=30 Nov 01 22:46:45.470031 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:45.469991 5011 patch_prober.go:29] interesting pod/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:46:45.470604 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:45.470549 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" podUID="58c1620e-7736-4fe3-bfde-8024cfb8d2e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:46:46.179639 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:46.179602 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm" Nov 01 22:46:46.752984 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:46.752947 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-4vmss"] Nov 01 22:46:46.962074 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:46:46.962039 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:46:49.120097 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.120059 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:46:49.224494 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.211882 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c0d076-9775f9644-dn545" Nov 01 22:46:49.224903 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.224817 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-workload-socket\") pod \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " Nov 01 22:46:49.225032 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.224928 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-workload-certs\") pod \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " Nov 01 22:46:49.225032 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.224970 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istiod-ca-cert\") pod \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " Nov 01 22:46:49.225032 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.225021 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-kube-api-access-bg8sl\" (UniqueName: \"kubernetes.io/projected/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-kube-api-access-bg8sl\") pod \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " Nov 01 22:46:49.225207 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.225075 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istio-podinfo\") pod \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " Nov 01 22:46:49.225207 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.225120 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-credential-socket\") pod \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " Nov 01 22:46:49.225207 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.225165 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istio-data\") pod \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " Nov 01 22:46:49.225369 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.225214 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istio-envoy\") pod \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " Nov 01 22:46:49.225453 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.225436 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istio-token\" (UniqueName: \"kubernetes.io/projected/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istio-token\") pod \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\" (UID: \"58c1620e-7736-4fe3-bfde-8024cfb8d2e5\") " Nov 01 22:46:49.226877 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.226830 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-credential-socket" pod "58c1620e-7736-4fe3-bfde-8024cfb8d2e5" (UID: "58c1620e-7736-4fe3-bfde-8024cfb8d2e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:49.231154 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.231104 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istio-token" pod "58c1620e-7736-4fe3-bfde-8024cfb8d2e5" (UID: "58c1620e-7736-4fe3-bfde-8024cfb8d2e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:49.231804 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.231742 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-workload-socket" pod "58c1620e-7736-4fe3-bfde-8024cfb8d2e5" (UID: "58c1620e-7736-4fe3-bfde-8024cfb8d2e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:49.231804 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.231765 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istiod-ca-cert" pod "58c1620e-7736-4fe3-bfde-8024cfb8d2e5" (UID: "58c1620e-7736-4fe3-bfde-8024cfb8d2e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:49.232050 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.231996 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istio-data" pod "58c1620e-7736-4fe3-bfde-8024cfb8d2e5" (UID: "58c1620e-7736-4fe3-bfde-8024cfb8d2e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:49.232636 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.232612 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-workload-certs" pod "58c1620e-7736-4fe3-bfde-8024cfb8d2e5" (UID: "58c1620e-7736-4fe3-bfde-8024cfb8d2e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:49.244636 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.243388 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istio-envoy" pod "58c1620e-7736-4fe3-bfde-8024cfb8d2e5" (UID: "58c1620e-7736-4fe3-bfde-8024cfb8d2e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:49.244636 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.244518 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istio-podinfo" pod "58c1620e-7736-4fe3-bfde-8024cfb8d2e5" (UID: "58c1620e-7736-4fe3-bfde-8024cfb8d2e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:46:49.248149 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.248121 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-kube-api-access-bg8sl" pod "58c1620e-7736-4fe3-bfde-8024cfb8d2e5" (UID: "58c1620e-7736-4fe3-bfde-8024cfb8d2e5"). InnerVolumeSpecName "kube-api-access-bg8sl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:49.326998 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.326903 5011 generic.go:356] "Generic (PLEG): container finished" podID="58c1620e-7736-4fe3-bfde-8024cfb8d2e5" containerID="52f591d9f3e3a4645a5195318b13b1b4272e64b6b0f5785d56cc7e5fc93e1747" exitCode=0 Nov 01 22:46:49.327190 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.327168 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" event={"ID":"58c1620e-7736-4fe3-bfde-8024cfb8d2e5","Type":"ContainerDied","Data":"52f591d9f3e3a4645a5195318b13b1b4272e64b6b0f5785d56cc7e5fc93e1747"} Nov 01 22:46:49.327260 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.327231 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" event={"ID":"58c1620e-7736-4fe3-bfde-8024cfb8d2e5","Type":"ContainerDied","Data":"c71a95b2fd179057e493b20b5f8472fb5c4bd6d7ab6731682aff357864444829"} Nov 01 22:46:49.327313 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.327266 5011 scope.go:122] "RemoveContainer" containerID="52f591d9f3e3a4645a5195318b13b1b4272e64b6b0f5785d56cc7e5fc93e1747" Nov 01 22:46:49.327461 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.327444 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r" Nov 01 22:46:49.327733 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.327712 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:49.328559 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.328535 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:49.328559 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.328561 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:49.328723 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.328575 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:49.328723 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.328587 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:49.328723 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.328600 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:49.328723 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.328612 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:49.328723 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.328625 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:49.328723 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.328638 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg8sl\" (UniqueName: \"kubernetes.io/projected/58c1620e-7736-4fe3-bfde-8024cfb8d2e5-kube-api-access-bg8sl\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:49.346629 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.346602 5011 scope.go:122] "RemoveContainer" containerID="dcce2c0ba8383a16bd9638aa080632c45f034cb3bc7780fbb9fcbd47213854f5" Nov 01 22:46:49.358854 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.358818 5011 scope.go:122] "RemoveContainer" containerID="52f591d9f3e3a4645a5195318b13b1b4272e64b6b0f5785d56cc7e5fc93e1747" Nov 01 22:46:49.359185 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:46:49.359166 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52f591d9f3e3a4645a5195318b13b1b4272e64b6b0f5785d56cc7e5fc93e1747\": container with ID starting with 52f591d9f3e3a4645a5195318b13b1b4272e64b6b0f5785d56cc7e5fc93e1747 not found: ID does not exist" containerID="52f591d9f3e3a4645a5195318b13b1b4272e64b6b0f5785d56cc7e5fc93e1747" Nov 01 22:46:49.359231 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.359197 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f591d9f3e3a4645a5195318b13b1b4272e64b6b0f5785d56cc7e5fc93e1747"} err="failed to get container status \"52f591d9f3e3a4645a5195318b13b1b4272e64b6b0f5785d56cc7e5fc93e1747\": rpc error: code = NotFound desc = could not find container \"52f591d9f3e3a4645a5195318b13b1b4272e64b6b0f5785d56cc7e5fc93e1747\": container with ID starting with 52f591d9f3e3a4645a5195318b13b1b4272e64b6b0f5785d56cc7e5fc93e1747 not found: ID does not exist" Nov 01 22:46:49.359231 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.359214 5011 scope.go:122] "RemoveContainer" containerID="dcce2c0ba8383a16bd9638aa080632c45f034cb3bc7780fbb9fcbd47213854f5" Nov 01 22:46:49.359532 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:46:49.359505 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcce2c0ba8383a16bd9638aa080632c45f034cb3bc7780fbb9fcbd47213854f5\": container with ID starting with dcce2c0ba8383a16bd9638aa080632c45f034cb3bc7780fbb9fcbd47213854f5 not found: ID does not exist" containerID="dcce2c0ba8383a16bd9638aa080632c45f034cb3bc7780fbb9fcbd47213854f5" Nov 01 22:46:49.359629 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.359541 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcce2c0ba8383a16bd9638aa080632c45f034cb3bc7780fbb9fcbd47213854f5"} err="failed to get container status \"dcce2c0ba8383a16bd9638aa080632c45f034cb3bc7780fbb9fcbd47213854f5\": rpc error: code = NotFound desc = could not find container \"dcce2c0ba8383a16bd9638aa080632c45f034cb3bc7780fbb9fcbd47213854f5\": container with ID starting with dcce2c0ba8383a16bd9638aa080632c45f034cb3bc7780fbb9fcbd47213854f5 not found: ID does not exist" Nov 01 22:46:49.374345 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.374315 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r"] Nov 01 22:46:49.382574 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.382542 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2a5c3234-openshift-default-7fbdf68d8f-q6l6r"] Nov 01 22:46:49.783859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:49.780347 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58c1620e-7736-4fe3-bfde-8024cfb8d2e5" path="/var/lib/kubelet/pods/58c1620e-7736-4fe3-bfde-8024cfb8d2e5/volumes" Nov 01 22:46:50.507536 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:50.507491 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b34688-798c5d88d5-vk4cj"] Nov 01 22:46:50.508344 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:50.508326 5011 cpu_manager.go:414] "containerMap: removing container" podUID="58c1620e-7736-4fe3-bfde-8024cfb8d2e5" containerName="istio-proxy" Nov 01 22:46:50.508407 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:50.508346 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58c1620e-7736-4fe3-bfde-8024cfb8d2e5" containerName="istio-proxy" Nov 01 22:46:50.508407 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:50.508361 5011 cpu_manager.go:414] "containerMap: removing container" podUID="58c1620e-7736-4fe3-bfde-8024cfb8d2e5" containerName="istio-proxy" Nov 01 22:46:50.508407 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:50.508366 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58c1620e-7736-4fe3-bfde-8024cfb8d2e5" containerName="istio-proxy" Nov 01 22:46:50.508508 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:50.508501 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="58c1620e-7736-4fe3-bfde-8024cfb8d2e5" containerName="istio-proxy" Nov 01 22:46:50.508542 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:50.508510 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="58c1620e-7736-4fe3-bfde-8024cfb8d2e5" containerName="istio-proxy" Nov 01 22:46:50.514240 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:50.514218 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b34688-798c5d88d5-vk4cj" Nov 01 22:46:50.541885 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:50.541595 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-b34688-dockercfg-wd2cp\"" Nov 01 22:46:50.548921 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:50.548895 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b34688-69c5567c75-sdw9t"] Nov 01 22:46:50.554533 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:50.554510 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b34688-798c5d88d5-vk4cj"] Nov 01 22:46:50.554690 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:50.554625 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34688-69c5567c75-sdw9t" Nov 01 22:46:50.593155 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:50.591409 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-b34688\"" Nov 01 22:46:50.786380 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:50.786286 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2273f99-763d-41a5-bcdd-af0cf9626a2d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b34688-798c5d88d5-vk4cj\" (UID: \"e2273f99-763d-41a5-bcdd-af0cf9626a2d\") " pod="openshift-ingress/payload-processing-e2e-api-a-b34688-798c5d88d5-vk4cj" Nov 01 22:46:50.786709 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:50.786684 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wxzn\" (UniqueName: \"kubernetes.io/projected/e2273f99-763d-41a5-bcdd-af0cf9626a2d-kube-api-access-9wxzn\") pod \"payload-processing-e2e-api-a-b34688-798c5d88d5-vk4cj\" (UID: \"e2273f99-763d-41a5-bcdd-af0cf9626a2d\") " pod="openshift-ingress/payload-processing-e2e-api-a-b34688-798c5d88d5-vk4cj" Nov 01 22:46:50.786895 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:50.786878 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e43bcf5-a6f3-4119-af1b-45fb66056b1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b34688-69c5567c75-sdw9t\" (UID: \"8e43bcf5-a6f3-4119-af1b-45fb66056b1a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34688-69c5567c75-sdw9t" Nov 01 22:46:50.787188 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:50.787167 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4ldf\" (UniqueName: \"kubernetes.io/projected/8e43bcf5-a6f3-4119-af1b-45fb66056b1a-kube-api-access-x4ldf\") pod \"payload-pre-processing-e2e-api-a-b34688-69c5567c75-sdw9t\" (UID: \"8e43bcf5-a6f3-4119-af1b-45fb66056b1a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34688-69c5567c75-sdw9t" Nov 01 22:46:50.909864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:50.901126 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4ldf\" (UniqueName: \"kubernetes.io/projected/8e43bcf5-a6f3-4119-af1b-45fb66056b1a-kube-api-access-x4ldf\") pod \"payload-pre-processing-e2e-api-a-b34688-69c5567c75-sdw9t\" (UID: \"8e43bcf5-a6f3-4119-af1b-45fb66056b1a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34688-69c5567c75-sdw9t" Nov 01 22:46:50.935461 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:50.935428 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2273f99-763d-41a5-bcdd-af0cf9626a2d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b34688-798c5d88d5-vk4cj\" (UID: \"e2273f99-763d-41a5-bcdd-af0cf9626a2d\") " pod="openshift-ingress/payload-processing-e2e-api-a-b34688-798c5d88d5-vk4cj" Nov 01 22:46:50.979215 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:50.979177 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wxzn\" (UniqueName: \"kubernetes.io/projected/e2273f99-763d-41a5-bcdd-af0cf9626a2d-kube-api-access-9wxzn\") pod \"payload-processing-e2e-api-a-b34688-798c5d88d5-vk4cj\" (UID: \"e2273f99-763d-41a5-bcdd-af0cf9626a2d\") " pod="openshift-ingress/payload-processing-e2e-api-a-b34688-798c5d88d5-vk4cj" Nov 01 22:46:50.980848 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:50.980806 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e43bcf5-a6f3-4119-af1b-45fb66056b1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b34688-69c5567c75-sdw9t\" (UID: \"8e43bcf5-a6f3-4119-af1b-45fb66056b1a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34688-69c5567c75-sdw9t" Nov 01 22:46:50.981035 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:50.947846 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2273f99-763d-41a5-bcdd-af0cf9626a2d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b34688-798c5d88d5-vk4cj\" (UID: \"e2273f99-763d-41a5-bcdd-af0cf9626a2d\") " pod="openshift-ingress/payload-processing-e2e-api-a-b34688-798c5d88d5-vk4cj" Nov 01 22:46:50.981143 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:50.965971 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b34688-69c5567c75-sdw9t"] Nov 01 22:46:50.990275 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:50.982091 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e43bcf5-a6f3-4119-af1b-45fb66056b1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b34688-69c5567c75-sdw9t\" (UID: \"8e43bcf5-a6f3-4119-af1b-45fb66056b1a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34688-69c5567c75-sdw9t" Nov 01 22:46:50.990904 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:50.990814 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4ldf\" (UniqueName: \"kubernetes.io/projected/8e43bcf5-a6f3-4119-af1b-45fb66056b1a-kube-api-access-x4ldf\") pod \"payload-pre-processing-e2e-api-a-b34688-69c5567c75-sdw9t\" (UID: \"8e43bcf5-a6f3-4119-af1b-45fb66056b1a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34688-69c5567c75-sdw9t" Nov 01 22:46:51.007139 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:51.001440 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wxzn\" (UniqueName: \"kubernetes.io/projected/e2273f99-763d-41a5-bcdd-af0cf9626a2d-kube-api-access-9wxzn\") pod \"payload-processing-e2e-api-a-b34688-798c5d88d5-vk4cj\" (UID: \"e2273f99-763d-41a5-bcdd-af0cf9626a2d\") " pod="openshift-ingress/payload-processing-e2e-api-a-b34688-798c5d88d5-vk4cj" Nov 01 22:46:51.128894 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:51.128800 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b34688-798c5d88d5-vk4cj" Nov 01 22:46:51.213786 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:51.213536 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34688-69c5567c75-sdw9t" Nov 01 22:46:51.418644 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:46:51.418616 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2273f99_763d_41a5_bcdd_af0cf9626a2d.slice/crio-1d61d5b6e2fc807b3921d63a5fee622bd5eddf715c2ce1c44915ec8ea8fde7ea WatchSource:0}: Error finding container 1d61d5b6e2fc807b3921d63a5fee622bd5eddf715c2ce1c44915ec8ea8fde7ea: Status 404 returned error can't find the container with id 1d61d5b6e2fc807b3921d63a5fee622bd5eddf715c2ce1c44915ec8ea8fde7ea Nov 01 22:46:51.419101 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:51.419078 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b34688-798c5d88d5-vk4cj"] Nov 01 22:46:51.533458 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:51.533424 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b34688-69c5567c75-sdw9t"] Nov 01 22:46:51.769857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:51.755321 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-4vmss" podUID="16f65edd-5dd3-4a6b-85e4-1fba8e61f207" containerName="payload-pre-processing" containerID="cri-o://2ac8069535645eb66d05f63237e6c40d5180ffb0a435d146a57f2042075a199e" gracePeriod=25 Nov 01 22:46:52.080165 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:52.080139 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-4vmss" Nov 01 22:46:52.111477 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:52.111444 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16f65edd-5dd3-4a6b-85e4-1fba8e61f207-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16f65edd-5dd3-4a6b-85e4-1fba8e61f207-plugins-config-volume\") pod \"16f65edd-5dd3-4a6b-85e4-1fba8e61f207\" (UID: \"16f65edd-5dd3-4a6b-85e4-1fba8e61f207\") " Nov 01 22:46:52.112419 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:52.111566 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16f65edd-5dd3-4a6b-85e4-1fba8e61f207-kube-api-access-htslt\" (UniqueName: \"kubernetes.io/projected/16f65edd-5dd3-4a6b-85e4-1fba8e61f207-kube-api-access-htslt\") pod \"16f65edd-5dd3-4a6b-85e4-1fba8e61f207\" (UID: \"16f65edd-5dd3-4a6b-85e4-1fba8e61f207\") " Nov 01 22:46:52.112419 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:52.112134 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16f65edd-5dd3-4a6b-85e4-1fba8e61f207-plugins-config-volume" pod "16f65edd-5dd3-4a6b-85e4-1fba8e61f207" (UID: "16f65edd-5dd3-4a6b-85e4-1fba8e61f207"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:52.112419 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:52.112368 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16f65edd-5dd3-4a6b-85e4-1fba8e61f207-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:52.114908 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:52.114878 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16f65edd-5dd3-4a6b-85e4-1fba8e61f207-kube-api-access-htslt" pod "16f65edd-5dd3-4a6b-85e4-1fba8e61f207" (UID: "16f65edd-5dd3-4a6b-85e4-1fba8e61f207"). InnerVolumeSpecName "kube-api-access-htslt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:52.188027 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:52.188002 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-947846-578fb595bf-l7kpp" Nov 01 22:46:52.214048 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:52.214017 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htslt\" (UniqueName: \"kubernetes.io/projected/16f65edd-5dd3-4a6b-85e4-1fba8e61f207-kube-api-access-htslt\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:52.357353 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:52.357264 5011 generic.go:356] "Generic (PLEG): container finished" podID="16f65edd-5dd3-4a6b-85e4-1fba8e61f207" containerID="2ac8069535645eb66d05f63237e6c40d5180ffb0a435d146a57f2042075a199e" exitCode=0 Nov 01 22:46:52.357353 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:52.357318 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-4vmss" Nov 01 22:46:52.357583 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:52.357353 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-4vmss" event={"ID":"16f65edd-5dd3-4a6b-85e4-1fba8e61f207","Type":"ContainerDied","Data":"2ac8069535645eb66d05f63237e6c40d5180ffb0a435d146a57f2042075a199e"} Nov 01 22:46:52.357583 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:52.357416 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-4vmss" event={"ID":"16f65edd-5dd3-4a6b-85e4-1fba8e61f207","Type":"ContainerDied","Data":"7cdf4091dd28d356e54691fdd3b68c90469db16c967678680019dcd3df959779"} Nov 01 22:46:52.357583 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:52.357454 5011 scope.go:122] "RemoveContainer" containerID="2ac8069535645eb66d05f63237e6c40d5180ffb0a435d146a57f2042075a199e" Nov 01 22:46:52.359552 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:52.359527 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34688-69c5567c75-sdw9t" event={"ID":"8e43bcf5-a6f3-4119-af1b-45fb66056b1a","Type":"ContainerStarted","Data":"da1b9d015c7fc48a4d5dc5e94521cb616d12d6660d2b29fcdc31c9ba35913a0d"} Nov 01 22:46:52.359670 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:52.359572 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34688-69c5567c75-sdw9t" event={"ID":"8e43bcf5-a6f3-4119-af1b-45fb66056b1a","Type":"ContainerStarted","Data":"d86a32282a34099692d25cec4efce8cfc17fde714f52ad93a69124e5be0ee4dc"} Nov 01 22:46:52.359670 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:52.359601 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34688-69c5567c75-sdw9t" Nov 01 22:46:52.361470 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:52.361446 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b34688-798c5d88d5-vk4cj" event={"ID":"e2273f99-763d-41a5-bcdd-af0cf9626a2d","Type":"ContainerStarted","Data":"de6691806d9e1505344e0d057f2203d1bf4a0155b522e3532b54372279bc88b3"} Nov 01 22:46:52.361597 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:52.361493 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-b34688-798c5d88d5-vk4cj" Nov 01 22:46:52.361597 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:52.361517 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b34688-798c5d88d5-vk4cj" event={"ID":"e2273f99-763d-41a5-bcdd-af0cf9626a2d","Type":"ContainerStarted","Data":"1d61d5b6e2fc807b3921d63a5fee622bd5eddf715c2ce1c44915ec8ea8fde7ea"} Nov 01 22:46:52.369811 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:52.369795 5011 scope.go:122] "RemoveContainer" containerID="2ac8069535645eb66d05f63237e6c40d5180ffb0a435d146a57f2042075a199e" Nov 01 22:46:52.370148 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:46:52.370130 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ac8069535645eb66d05f63237e6c40d5180ffb0a435d146a57f2042075a199e\": container with ID starting with 2ac8069535645eb66d05f63237e6c40d5180ffb0a435d146a57f2042075a199e not found: ID does not exist" containerID="2ac8069535645eb66d05f63237e6c40d5180ffb0a435d146a57f2042075a199e" Nov 01 22:46:52.370205 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:52.370155 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ac8069535645eb66d05f63237e6c40d5180ffb0a435d146a57f2042075a199e"} err="failed to get container status \"2ac8069535645eb66d05f63237e6c40d5180ffb0a435d146a57f2042075a199e\": rpc error: code = NotFound desc = could not find container \"2ac8069535645eb66d05f63237e6c40d5180ffb0a435d146a57f2042075a199e\": container with ID starting with 2ac8069535645eb66d05f63237e6c40d5180ffb0a435d146a57f2042075a199e not found: ID does not exist" Nov 01 22:46:52.392232 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:52.392187 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34688-69c5567c75-sdw9t" podStartSLOduration=2.392173139 podStartE2EDuration="2.392173139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:52.390150527 +0000 UTC m=+1380.794741163" watchObservedRunningTime="2026-11-01 22:46:52.392173139 +0000 UTC m=+1380.796763815" Nov 01 22:46:52.414057 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:52.414004 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-b34688-798c5d88d5-vk4cj" podStartSLOduration=2.413989859 podStartE2EDuration="2.413989859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:52.412680792 +0000 UTC m=+1380.817271425" watchObservedRunningTime="2026-11-01 22:46:52.413989859 +0000 UTC m=+1380.818580490" Nov 01 22:46:52.447671 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:52.447640 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-4vmss"] Nov 01 22:46:52.451397 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:52.451374 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-4vmss"] Nov 01 22:46:53.199013 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:53.198982 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-947846-7965f5db47-fr9tf" Nov 01 22:46:53.749176 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:53.749143 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16f65edd-5dd3-4a6b-85e4-1fba8e61f207" path="/var/lib/kubelet/pods/16f65edd-5dd3-4a6b-85e4-1fba8e61f207/volumes" Nov 01 22:46:56.932875 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:56.932822 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z"] Nov 01 22:46:56.935887 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:56.933813 5011 cpu_manager.go:414] "containerMap: removing container" podUID="16f65edd-5dd3-4a6b-85e4-1fba8e61f207" containerName="payload-pre-processing" Nov 01 22:46:56.936049 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:56.935888 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16f65edd-5dd3-4a6b-85e4-1fba8e61f207" containerName="payload-pre-processing" Nov 01 22:46:56.936205 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:56.936092 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="16f65edd-5dd3-4a6b-85e4-1fba8e61f207" containerName="payload-pre-processing" Nov 01 22:46:56.941529 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:56.941504 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z" Nov 01 22:46:56.949094 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:56.949066 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-947846-serving-cert\"" Nov 01 22:46:56.964876 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:56.953309 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z"] Nov 01 22:46:57.022427 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:57.022390 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f94602ac-6d0b-4d6e-a4e6-dd99179ec535-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z\" (UID: \"f94602ac-6d0b-4d6e-a4e6-dd99179ec535\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z" Nov 01 22:46:57.022624 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:57.022520 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f94602ac-6d0b-4d6e-a4e6-dd99179ec535-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z\" (UID: \"f94602ac-6d0b-4d6e-a4e6-dd99179ec535\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z" Nov 01 22:46:57.022725 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:57.022700 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f94602ac-6d0b-4d6e-a4e6-dd99179ec535-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z\" (UID: \"f94602ac-6d0b-4d6e-a4e6-dd99179ec535\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z" Nov 01 22:46:57.022863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:57.022844 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78ns4\" (UniqueName: \"kubernetes.io/projected/f94602ac-6d0b-4d6e-a4e6-dd99179ec535-kube-api-access-78ns4\") pod \"maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z\" (UID: \"f94602ac-6d0b-4d6e-a4e6-dd99179ec535\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z" Nov 01 22:46:57.130382 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:57.129427 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f94602ac-6d0b-4d6e-a4e6-dd99179ec535-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z\" (UID: \"f94602ac-6d0b-4d6e-a4e6-dd99179ec535\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z" Nov 01 22:46:57.130382 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:57.129615 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f94602ac-6d0b-4d6e-a4e6-dd99179ec535-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z\" (UID: \"f94602ac-6d0b-4d6e-a4e6-dd99179ec535\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z" Nov 01 22:46:57.130382 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:57.129757 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78ns4\" (UniqueName: \"kubernetes.io/projected/f94602ac-6d0b-4d6e-a4e6-dd99179ec535-kube-api-access-78ns4\") pod \"maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z\" (UID: \"f94602ac-6d0b-4d6e-a4e6-dd99179ec535\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z" Nov 01 22:46:57.130382 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:57.130031 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f94602ac-6d0b-4d6e-a4e6-dd99179ec535-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z\" (UID: \"f94602ac-6d0b-4d6e-a4e6-dd99179ec535\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z" Nov 01 22:46:57.133749 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:57.133716 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f94602ac-6d0b-4d6e-a4e6-dd99179ec535-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z\" (UID: \"f94602ac-6d0b-4d6e-a4e6-dd99179ec535\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z" Nov 01 22:46:57.135820 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:57.134410 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f94602ac-6d0b-4d6e-a4e6-dd99179ec535-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z\" (UID: \"f94602ac-6d0b-4d6e-a4e6-dd99179ec535\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z" Nov 01 22:46:57.151895 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:57.137611 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f94602ac-6d0b-4d6e-a4e6-dd99179ec535-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z\" (UID: \"f94602ac-6d0b-4d6e-a4e6-dd99179ec535\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z" Nov 01 22:46:57.184244 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:57.184154 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78ns4\" (UniqueName: \"kubernetes.io/projected/f94602ac-6d0b-4d6e-a4e6-dd99179ec535-kube-api-access-78ns4\") pod \"maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z\" (UID: \"f94602ac-6d0b-4d6e-a4e6-dd99179ec535\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z" Nov 01 22:46:57.276583 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:57.276544 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z" Nov 01 22:46:57.569204 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:46:57.569173 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf94602ac_6d0b_4d6e_a4e6_dd99179ec535.slice/crio-fc24015b254d9365f8b602ea18f50860875033de6168f768f256def7ed41051c WatchSource:0}: Error finding container fc24015b254d9365f8b602ea18f50860875033de6168f768f256def7ed41051c: Status 404 returned error can't find the container with id fc24015b254d9365f8b602ea18f50860875033de6168f768f256def7ed41051c Nov 01 22:46:57.569455 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:57.569428 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z"] Nov 01 22:46:58.440575 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:58.440535 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z" event={"ID":"f94602ac-6d0b-4d6e-a4e6-dd99179ec535","Type":"ContainerStarted","Data":"bfe614735ede9d9a4fb23b29c2b69251d2f3618f68e6c0c81e1cdf7ec9d30eb4"} Nov 01 22:46:58.441004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:58.440600 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z" event={"ID":"f94602ac-6d0b-4d6e-a4e6-dd99179ec535","Type":"ContainerStarted","Data":"fc24015b254d9365f8b602ea18f50860875033de6168f768f256def7ed41051c"} Nov 01 22:46:58.441004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:58.440629 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z" Nov 01 22:46:58.467803 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:46:58.467747 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z" podStartSLOduration=1.973011159 podStartE2EDuration="2.467732902s" totalImagesPullingTime="494.721743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:58.463819615 +0000 UTC m=+1386.868410247" watchObservedRunningTime="2026-11-01 22:46:58.467732902 +0000 UTC m=+1386.872323535" Nov 01 22:47:03.372788 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:03.372757 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-b34688-798c5d88d5-vk4cj" Nov 01 22:47:03.373308 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:03.372866 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34688-69c5567c75-sdw9t" Nov 01 22:47:04.455099 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:04.455069 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z" Nov 01 22:47:08.044024 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:08.043981 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b34688-7b88d575b8-qb72d"] Nov 01 22:47:08.053192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:08.053162 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34688-7b88d575b8-qb72d" Nov 01 22:47:08.067731 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:08.067700 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-b34688-serving-cert\"" Nov 01 22:47:08.073555 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:08.073516 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b34688-7b88d575b8-qb72d"] Nov 01 22:47:08.133333 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:08.133297 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd9c2fec-dd43-44cc-9a0e-86cf68e5c915-metrics-certs\") pod \"maas-api-e2e-api-a-b34688-7b88d575b8-qb72d\" (UID: \"bd9c2fec-dd43-44cc-9a0e-86cf68e5c915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34688-7b88d575b8-qb72d" Nov 01 22:47:08.133514 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:08.133354 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd9c2fec-dd43-44cc-9a0e-86cf68e5c915-maas-api-tls\") pod \"maas-api-e2e-api-a-b34688-7b88d575b8-qb72d\" (UID: \"bd9c2fec-dd43-44cc-9a0e-86cf68e5c915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34688-7b88d575b8-qb72d" Nov 01 22:47:08.133514 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:08.133441 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tsmg\" (UniqueName: \"kubernetes.io/projected/bd9c2fec-dd43-44cc-9a0e-86cf68e5c915-kube-api-access-8tsmg\") pod \"maas-api-e2e-api-a-b34688-7b88d575b8-qb72d\" (UID: \"bd9c2fec-dd43-44cc-9a0e-86cf68e5c915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34688-7b88d575b8-qb72d" Nov 01 22:47:08.133594 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:08.133563 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd9c2fec-dd43-44cc-9a0e-86cf68e5c915-openshift-service-ca\") pod \"maas-api-e2e-api-a-b34688-7b88d575b8-qb72d\" (UID: \"bd9c2fec-dd43-44cc-9a0e-86cf68e5c915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34688-7b88d575b8-qb72d" Nov 01 22:47:08.258608 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:08.258241 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd9c2fec-dd43-44cc-9a0e-86cf68e5c915-metrics-certs\") pod \"maas-api-e2e-api-a-b34688-7b88d575b8-qb72d\" (UID: \"bd9c2fec-dd43-44cc-9a0e-86cf68e5c915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34688-7b88d575b8-qb72d" Nov 01 22:47:08.258608 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:08.258331 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd9c2fec-dd43-44cc-9a0e-86cf68e5c915-maas-api-tls\") pod \"maas-api-e2e-api-a-b34688-7b88d575b8-qb72d\" (UID: \"bd9c2fec-dd43-44cc-9a0e-86cf68e5c915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34688-7b88d575b8-qb72d" Nov 01 22:47:08.258608 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:08.258447 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tsmg\" (UniqueName: \"kubernetes.io/projected/bd9c2fec-dd43-44cc-9a0e-86cf68e5c915-kube-api-access-8tsmg\") pod \"maas-api-e2e-api-a-b34688-7b88d575b8-qb72d\" (UID: \"bd9c2fec-dd43-44cc-9a0e-86cf68e5c915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34688-7b88d575b8-qb72d" Nov 01 22:47:08.258608 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:08.258562 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd9c2fec-dd43-44cc-9a0e-86cf68e5c915-openshift-service-ca\") pod \"maas-api-e2e-api-a-b34688-7b88d575b8-qb72d\" (UID: \"bd9c2fec-dd43-44cc-9a0e-86cf68e5c915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34688-7b88d575b8-qb72d" Nov 01 22:47:08.261013 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:08.259615 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd9c2fec-dd43-44cc-9a0e-86cf68e5c915-openshift-service-ca\") pod \"maas-api-e2e-api-a-b34688-7b88d575b8-qb72d\" (UID: \"bd9c2fec-dd43-44cc-9a0e-86cf68e5c915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34688-7b88d575b8-qb72d" Nov 01 22:47:08.370117 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:08.365713 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd9c2fec-dd43-44cc-9a0e-86cf68e5c915-maas-api-tls\") pod \"maas-api-e2e-api-a-b34688-7b88d575b8-qb72d\" (UID: \"bd9c2fec-dd43-44cc-9a0e-86cf68e5c915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34688-7b88d575b8-qb72d" Nov 01 22:47:08.390105 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:08.390027 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tsmg\" (UniqueName: \"kubernetes.io/projected/bd9c2fec-dd43-44cc-9a0e-86cf68e5c915-kube-api-access-8tsmg\") pod \"maas-api-e2e-api-a-b34688-7b88d575b8-qb72d\" (UID: \"bd9c2fec-dd43-44cc-9a0e-86cf68e5c915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34688-7b88d575b8-qb72d" Nov 01 22:47:08.394504 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:08.394473 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd9c2fec-dd43-44cc-9a0e-86cf68e5c915-metrics-certs\") pod \"maas-api-e2e-api-a-b34688-7b88d575b8-qb72d\" (UID: \"bd9c2fec-dd43-44cc-9a0e-86cf68e5c915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34688-7b88d575b8-qb72d" Nov 01 22:47:08.691797 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:08.690115 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34688-7b88d575b8-qb72d" Nov 01 22:47:09.095972 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:09.095938 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b34688-7b88d575b8-qb72d"] Nov 01 22:47:09.120861 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:47:09.120774 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd9c2fec_dd43_44cc_9a0e_86cf68e5c915.slice/crio-d72ed78b5989343083aad0d05067c080bdcf885dd12b4aa2a69729a3f1052117 WatchSource:0}: Error finding container d72ed78b5989343083aad0d05067c080bdcf885dd12b4aa2a69729a3f1052117: Status 404 returned error can't find the container with id d72ed78b5989343083aad0d05067c080bdcf885dd12b4aa2a69729a3f1052117 Nov 01 22:47:09.568747 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:09.568703 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34688-7b88d575b8-qb72d" event={"ID":"bd9c2fec-dd43-44cc-9a0e-86cf68e5c915","Type":"ContainerStarted","Data":"d72ed78b5989343083aad0d05067c080bdcf885dd12b4aa2a69729a3f1052117"} Nov 01 22:47:10.578495 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:10.578442 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34688-7b88d575b8-qb72d" event={"ID":"bd9c2fec-dd43-44cc-9a0e-86cf68e5c915","Type":"ContainerStarted","Data":"fdfd3e100d969dad629e3740d2b8c680c29c4fff29de1269140feef25822cfa5"} Nov 01 22:47:10.579042 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:10.578543 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34688-7b88d575b8-qb72d" Nov 01 22:47:10.610320 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:10.610271 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34688-7b88d575b8-qb72d" podStartSLOduration=2.077409005 podStartE2EDuration="2.610256783s" totalImagesPullingTime="532.847778ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:47:10.606702128 +0000 UTC m=+1399.011292779" watchObservedRunningTime="2026-11-01 22:47:10.610256783 +0000 UTC m=+1399.014847415" Nov 01 22:47:12.675291 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:12.675259 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e202e7-58d7c8ff6c-6gzzg"] Nov 01 22:47:12.682157 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:12.682133 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e202e7-58d7c8ff6c-6gzzg" Nov 01 22:47:12.690287 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:12.690116 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e202e7\"" Nov 01 22:47:12.690947 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:12.690922 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e202e7-dockercfg-69zm6\"" Nov 01 22:47:12.697152 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:12.697122 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e202e7-58d7c8ff6c-6gzzg"] Nov 01 22:47:12.719526 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:12.719488 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c8ba82e-675b-48ee-bc33-d683b2c171cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e202e7-58d7c8ff6c-6gzzg\" (UID: \"1c8ba82e-675b-48ee-bc33-d683b2c171cc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e202e7-58d7c8ff6c-6gzzg" Nov 01 22:47:12.719695 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:12.719596 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlkgl\" (UniqueName: \"kubernetes.io/projected/1c8ba82e-675b-48ee-bc33-d683b2c171cc-kube-api-access-wlkgl\") pod \"payload-pre-processing-e2e-inf-a-e202e7-58d7c8ff6c-6gzzg\" (UID: \"1c8ba82e-675b-48ee-bc33-d683b2c171cc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e202e7-58d7c8ff6c-6gzzg" Nov 01 22:47:12.725105 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:12.725075 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e202e7-595b687479-zhkww"] Nov 01 22:47:12.731373 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:12.731347 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e202e7-595b687479-zhkww" Nov 01 22:47:12.733481 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:12.733454 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e202e7-595b687479-zhkww"] Nov 01 22:47:12.889857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:12.861695 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c8ba82e-675b-48ee-bc33-d683b2c171cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e202e7-58d7c8ff6c-6gzzg\" (UID: \"1c8ba82e-675b-48ee-bc33-d683b2c171cc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e202e7-58d7c8ff6c-6gzzg" Nov 01 22:47:12.889857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:12.861798 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlkgl\" (UniqueName: \"kubernetes.io/projected/1c8ba82e-675b-48ee-bc33-d683b2c171cc-kube-api-access-wlkgl\") pod \"payload-pre-processing-e2e-inf-a-e202e7-58d7c8ff6c-6gzzg\" (UID: \"1c8ba82e-675b-48ee-bc33-d683b2c171cc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e202e7-58d7c8ff6c-6gzzg" Nov 01 22:47:13.020909 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:13.019277 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vjhw\" (UniqueName: \"kubernetes.io/projected/583600ad-f7bb-49e2-898c-2bd002a6b1d6-kube-api-access-9vjhw\") pod \"payload-processing-e2e-inf-a-e202e7-595b687479-zhkww\" (UID: \"583600ad-f7bb-49e2-898c-2bd002a6b1d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e202e7-595b687479-zhkww" Nov 01 22:47:13.020909 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:13.019442 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/583600ad-f7bb-49e2-898c-2bd002a6b1d6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e202e7-595b687479-zhkww\" (UID: \"583600ad-f7bb-49e2-898c-2bd002a6b1d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e202e7-595b687479-zhkww" Nov 01 22:47:13.045039 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:13.045002 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c8ba82e-675b-48ee-bc33-d683b2c171cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e202e7-58d7c8ff6c-6gzzg\" (UID: \"1c8ba82e-675b-48ee-bc33-d683b2c171cc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e202e7-58d7c8ff6c-6gzzg" Nov 01 22:47:13.093857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:13.093331 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlkgl\" (UniqueName: \"kubernetes.io/projected/1c8ba82e-675b-48ee-bc33-d683b2c171cc-kube-api-access-wlkgl\") pod \"payload-pre-processing-e2e-inf-a-e202e7-58d7c8ff6c-6gzzg\" (UID: \"1c8ba82e-675b-48ee-bc33-d683b2c171cc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e202e7-58d7c8ff6c-6gzzg" Nov 01 22:47:13.163857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:13.158050 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vjhw\" (UniqueName: \"kubernetes.io/projected/583600ad-f7bb-49e2-898c-2bd002a6b1d6-kube-api-access-9vjhw\") pod \"payload-processing-e2e-inf-a-e202e7-595b687479-zhkww\" (UID: \"583600ad-f7bb-49e2-898c-2bd002a6b1d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e202e7-595b687479-zhkww" Nov 01 22:47:13.163857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:13.158117 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/583600ad-f7bb-49e2-898c-2bd002a6b1d6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e202e7-595b687479-zhkww\" (UID: \"583600ad-f7bb-49e2-898c-2bd002a6b1d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e202e7-595b687479-zhkww" Nov 01 22:47:13.163857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:13.159563 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/583600ad-f7bb-49e2-898c-2bd002a6b1d6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e202e7-595b687479-zhkww\" (UID: \"583600ad-f7bb-49e2-898c-2bd002a6b1d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e202e7-595b687479-zhkww" Nov 01 22:47:13.194350 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:13.194279 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vjhw\" (UniqueName: \"kubernetes.io/projected/583600ad-f7bb-49e2-898c-2bd002a6b1d6-kube-api-access-9vjhw\") pod \"payload-processing-e2e-inf-a-e202e7-595b687479-zhkww\" (UID: \"583600ad-f7bb-49e2-898c-2bd002a6b1d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e202e7-595b687479-zhkww" Nov 01 22:47:13.304808 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:13.304721 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e202e7-58d7c8ff6c-6gzzg" Nov 01 22:47:13.351479 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:13.351445 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e202e7-595b687479-zhkww" Nov 01 22:47:13.594413 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:47:13.594382 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8ba82e_675b_48ee_bc33_d683b2c171cc.slice/crio-5306e156bc29a2ea7ab6e04df14b76a3c802580539126397d48b9f9b0edb6459 WatchSource:0}: Error finding container 5306e156bc29a2ea7ab6e04df14b76a3c802580539126397d48b9f9b0edb6459: Status 404 returned error can't find the container with id 5306e156bc29a2ea7ab6e04df14b76a3c802580539126397d48b9f9b0edb6459 Nov 01 22:47:13.595171 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:13.595148 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e202e7-58d7c8ff6c-6gzzg"] Nov 01 22:47:13.604218 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:13.604194 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e202e7-58d7c8ff6c-6gzzg" event={"ID":"1c8ba82e-675b-48ee-bc33-d683b2c171cc","Type":"ContainerStarted","Data":"5306e156bc29a2ea7ab6e04df14b76a3c802580539126397d48b9f9b0edb6459"} Nov 01 22:47:13.667303 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:47:13.667270 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod583600ad_f7bb_49e2_898c_2bd002a6b1d6.slice/crio-771e46ad54e5189a338ffa7bfdc39675d7417b2ca1afb75d5db937afbd8d3bb4 WatchSource:0}: Error finding container 771e46ad54e5189a338ffa7bfdc39675d7417b2ca1afb75d5db937afbd8d3bb4: Status 404 returned error can't find the container with id 771e46ad54e5189a338ffa7bfdc39675d7417b2ca1afb75d5db937afbd8d3bb4 Nov 01 22:47:13.667862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:13.667815 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e202e7-595b687479-zhkww"] Nov 01 22:47:14.613491 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:14.613450 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e202e7-58d7c8ff6c-6gzzg" event={"ID":"1c8ba82e-675b-48ee-bc33-d683b2c171cc","Type":"ContainerStarted","Data":"7fecf81348812edb1f021717a7622a386b9297d316895a4793d14f79f4219ab0"} Nov 01 22:47:14.614499 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:14.614481 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e202e7-58d7c8ff6c-6gzzg" Nov 01 22:47:14.616285 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:14.616261 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e202e7-595b687479-zhkww" event={"ID":"583600ad-f7bb-49e2-898c-2bd002a6b1d6","Type":"ContainerStarted","Data":"51049bc1781caaff785e307ac425ad2c6a60d8002243726b388938e15978393b"} Nov 01 22:47:14.616398 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:14.616305 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e202e7-595b687479-zhkww" Nov 01 22:47:14.616398 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:14.616322 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e202e7-595b687479-zhkww" event={"ID":"583600ad-f7bb-49e2-898c-2bd002a6b1d6","Type":"ContainerStarted","Data":"771e46ad54e5189a338ffa7bfdc39675d7417b2ca1afb75d5db937afbd8d3bb4"} Nov 01 22:47:14.634509 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:14.634457 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e202e7-58d7c8ff6c-6gzzg" podStartSLOduration=2.634444136 podStartE2EDuration="2.634444136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:47:14.631913581 +0000 UTC m=+1403.036504214" watchObservedRunningTime="2026-11-01 22:47:14.634444136 +0000 UTC m=+1403.039034768" Nov 01 22:47:14.657485 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:14.657434 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-e202e7-595b687479-zhkww" podStartSLOduration=2.6574202 podStartE2EDuration="2.6574202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:47:14.653775283 +0000 UTC m=+1403.058365916" watchObservedRunningTime="2026-11-01 22:47:14.6574202 +0000 UTC m=+1403.062010845" Nov 01 22:47:16.624663 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:16.624633 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34688-7b88d575b8-qb72d" Nov 01 22:47:16.962182 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:47:16.962140 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:47:18.322441 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:18.322410 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2476f8-7564448745-xnlhh"] Nov 01 22:47:18.339001 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:18.338968 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2476f8-7f7457c99-8f7vl"] Nov 01 22:47:23.323331 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:23.323250 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-2476f8-7564448745-xnlhh" podUID="a5697b6f-7e9f-4705-8f34-698f2a5447ac" containerName="payload-processing" containerID="cri-o://44de70ac0415c04b052a40e222a28254c9c3fe2c23e7ac3abf67270b0c0f62a8" gracePeriod=25 Nov 01 22:47:23.339997 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:23.339961 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2476f8-7f7457c99-8f7vl" podUID="0e895bf2-3149-4322-a026-b2d1c72ca948" containerName="payload-pre-processing" containerID="cri-o://47181381fff81e95319c4eb3934eb5654400ed92050d3357c8fb911d58a2619e" gracePeriod=25 Nov 01 22:47:23.900390 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:23.900260 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2476f8-7564448745-xnlhh" Nov 01 22:47:23.907406 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:23.907373 5011 generic.go:356] "Generic (PLEG): container finished" podID="0e895bf2-3149-4322-a026-b2d1c72ca948" containerID="47181381fff81e95319c4eb3934eb5654400ed92050d3357c8fb911d58a2619e" exitCode=0 Nov 01 22:47:23.907561 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:23.907542 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2476f8-7f7457c99-8f7vl" event={"ID":"0e895bf2-3149-4322-a026-b2d1c72ca948","Type":"ContainerDied","Data":"47181381fff81e95319c4eb3934eb5654400ed92050d3357c8fb911d58a2619e"} Nov 01 22:47:23.909194 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:23.909172 5011 generic.go:356] "Generic (PLEG): container finished" podID="a5697b6f-7e9f-4705-8f34-698f2a5447ac" containerID="44de70ac0415c04b052a40e222a28254c9c3fe2c23e7ac3abf67270b0c0f62a8" exitCode=0 Nov 01 22:47:23.909294 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:23.909279 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2476f8-7564448745-xnlhh" event={"ID":"a5697b6f-7e9f-4705-8f34-698f2a5447ac","Type":"ContainerDied","Data":"44de70ac0415c04b052a40e222a28254c9c3fe2c23e7ac3abf67270b0c0f62a8"} Nov 01 22:47:23.909336 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:23.909323 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2476f8-7564448745-xnlhh" event={"ID":"a5697b6f-7e9f-4705-8f34-698f2a5447ac","Type":"ContainerDied","Data":"6c0669f74c6e8b4feee10f3dc3b50f71bd4b68c8d48467e704e0817811b7a29c"} Nov 01 22:47:23.909367 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:23.909351 5011 scope.go:122] "RemoveContainer" containerID="44de70ac0415c04b052a40e222a28254c9c3fe2c23e7ac3abf67270b0c0f62a8" Nov 01 22:47:23.909520 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:23.909505 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2476f8-7564448745-xnlhh" Nov 01 22:47:23.929560 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:23.929530 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2476f8-7f7457c99-8f7vl" Nov 01 22:47:23.931205 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:23.930381 5011 scope.go:122] "RemoveContainer" containerID="44de70ac0415c04b052a40e222a28254c9c3fe2c23e7ac3abf67270b0c0f62a8" Nov 01 22:47:23.931205 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:47:23.930674 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44de70ac0415c04b052a40e222a28254c9c3fe2c23e7ac3abf67270b0c0f62a8\": container with ID starting with 44de70ac0415c04b052a40e222a28254c9c3fe2c23e7ac3abf67270b0c0f62a8 not found: ID does not exist" containerID="44de70ac0415c04b052a40e222a28254c9c3fe2c23e7ac3abf67270b0c0f62a8" Nov 01 22:47:23.931205 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:23.930708 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44de70ac0415c04b052a40e222a28254c9c3fe2c23e7ac3abf67270b0c0f62a8"} err="failed to get container status \"44de70ac0415c04b052a40e222a28254c9c3fe2c23e7ac3abf67270b0c0f62a8\": rpc error: code = NotFound desc = could not find container \"44de70ac0415c04b052a40e222a28254c9c3fe2c23e7ac3abf67270b0c0f62a8\": container with ID starting with 44de70ac0415c04b052a40e222a28254c9c3fe2c23e7ac3abf67270b0c0f62a8 not found: ID does not exist" Nov 01 22:47:23.988688 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:23.988573 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5697b6f-7e9f-4705-8f34-698f2a5447ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5697b6f-7e9f-4705-8f34-698f2a5447ac-plugins-config-volume\") pod \"a5697b6f-7e9f-4705-8f34-698f2a5447ac\" (UID: \"a5697b6f-7e9f-4705-8f34-698f2a5447ac\") " Nov 01 22:47:23.988914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:23.988734 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5697b6f-7e9f-4705-8f34-698f2a5447ac-kube-api-access-55rk4\" (UniqueName: \"kubernetes.io/projected/a5697b6f-7e9f-4705-8f34-698f2a5447ac-kube-api-access-55rk4\") pod \"a5697b6f-7e9f-4705-8f34-698f2a5447ac\" (UID: \"a5697b6f-7e9f-4705-8f34-698f2a5447ac\") " Nov 01 22:47:23.989438 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:23.989383 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5697b6f-7e9f-4705-8f34-698f2a5447ac-plugins-config-volume" pod "a5697b6f-7e9f-4705-8f34-698f2a5447ac" (UID: "a5697b6f-7e9f-4705-8f34-698f2a5447ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:23.990062 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:23.990037 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5697b6f-7e9f-4705-8f34-698f2a5447ac-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:24.022121 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:24.020863 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5697b6f-7e9f-4705-8f34-698f2a5447ac-kube-api-access-55rk4" pod "a5697b6f-7e9f-4705-8f34-698f2a5447ac" (UID: "a5697b6f-7e9f-4705-8f34-698f2a5447ac"). InnerVolumeSpecName "kube-api-access-55rk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:24.092874 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:24.091202 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e895bf2-3149-4322-a026-b2d1c72ca948-kube-api-access-7qvvr\" (UniqueName: \"kubernetes.io/projected/0e895bf2-3149-4322-a026-b2d1c72ca948-kube-api-access-7qvvr\") pod \"0e895bf2-3149-4322-a026-b2d1c72ca948\" (UID: \"0e895bf2-3149-4322-a026-b2d1c72ca948\") " Nov 01 22:47:24.092874 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:24.092132 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e895bf2-3149-4322-a026-b2d1c72ca948-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e895bf2-3149-4322-a026-b2d1c72ca948-plugins-config-volume\") pod \"0e895bf2-3149-4322-a026-b2d1c72ca948\" (UID: \"0e895bf2-3149-4322-a026-b2d1c72ca948\") " Nov 01 22:47:24.092874 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:24.092588 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e895bf2-3149-4322-a026-b2d1c72ca948-plugins-config-volume" pod "0e895bf2-3149-4322-a026-b2d1c72ca948" (UID: "0e895bf2-3149-4322-a026-b2d1c72ca948"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:24.093476 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:24.093450 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e895bf2-3149-4322-a026-b2d1c72ca948-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:24.093580 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:24.093480 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55rk4\" (UniqueName: \"kubernetes.io/projected/a5697b6f-7e9f-4705-8f34-698f2a5447ac-kube-api-access-55rk4\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:24.093894 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:24.093871 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e895bf2-3149-4322-a026-b2d1c72ca948-kube-api-access-7qvvr" pod "0e895bf2-3149-4322-a026-b2d1c72ca948" (UID: "0e895bf2-3149-4322-a026-b2d1c72ca948"). InnerVolumeSpecName "kube-api-access-7qvvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:24.195283 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:24.195246 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qvvr\" (UniqueName: \"kubernetes.io/projected/0e895bf2-3149-4322-a026-b2d1c72ca948-kube-api-access-7qvvr\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:24.275858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:24.269198 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2476f8-7564448745-xnlhh"] Nov 01 22:47:24.279391 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:24.279363 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2476f8-7564448745-xnlhh"] Nov 01 22:47:24.922438 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:24.922403 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2476f8-7f7457c99-8f7vl" event={"ID":"0e895bf2-3149-4322-a026-b2d1c72ca948","Type":"ContainerDied","Data":"f95551a633e042ee3f4d54a10e15e6b4a82594e3ff3a0463b55d98245e77a77c"} Nov 01 22:47:24.923005 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:24.922441 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2476f8-7f7457c99-8f7vl" Nov 01 22:47:24.923005 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:24.922463 5011 scope.go:122] "RemoveContainer" containerID="47181381fff81e95319c4eb3934eb5654400ed92050d3357c8fb911d58a2619e" Nov 01 22:47:24.976306 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:24.976270 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2476f8-7f7457c99-8f7vl"] Nov 01 22:47:24.980989 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:24.980962 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2476f8-7f7457c99-8f7vl"] Nov 01 22:47:25.646857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:25.637478 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e202e7-595b687479-zhkww" Nov 01 22:47:25.935915 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:25.933469 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e895bf2-3149-4322-a026-b2d1c72ca948" path="/var/lib/kubelet/pods/0e895bf2-3149-4322-a026-b2d1c72ca948/volumes" Nov 01 22:47:25.987896 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:25.949976 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5697b6f-7e9f-4705-8f34-698f2a5447ac" path="/var/lib/kubelet/pods/a5697b6f-7e9f-4705-8f34-698f2a5447ac/volumes" Nov 01 22:47:26.636630 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:26.636601 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e202e7-58d7c8ff6c-6gzzg" Nov 01 22:47:27.487949 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:27.486346 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx"] Nov 01 22:47:27.488473 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:27.488128 5011 cpu_manager.go:414] "containerMap: removing container" podUID="0e895bf2-3149-4322-a026-b2d1c72ca948" containerName="payload-pre-processing" Nov 01 22:47:27.488473 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:27.488172 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e895bf2-3149-4322-a026-b2d1c72ca948" containerName="payload-pre-processing" Nov 01 22:47:27.488473 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:27.488212 5011 cpu_manager.go:414] "containerMap: removing container" podUID="a5697b6f-7e9f-4705-8f34-698f2a5447ac" containerName="payload-processing" Nov 01 22:47:27.488473 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:27.488223 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5697b6f-7e9f-4705-8f34-698f2a5447ac" containerName="payload-processing" Nov 01 22:47:27.488698 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:27.488507 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5697b6f-7e9f-4705-8f34-698f2a5447ac" containerName="payload-processing" Nov 01 22:47:27.488698 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:27.488549 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e895bf2-3149-4322-a026-b2d1c72ca948" containerName="payload-pre-processing" Nov 01 22:47:27.494823 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:27.494799 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx"] Nov 01 22:47:27.495092 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:27.494950 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx" Nov 01 22:47:27.501165 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:27.501013 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-44a9aa-serving-cert\"" Nov 01 22:47:27.536459 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:27.536423 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1059eeab-0cdf-47fe-997c-71245e8407a2-maas-api-tls\") pod \"maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx\" (UID: \"1059eeab-0cdf-47fe-997c-71245e8407a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx" Nov 01 22:47:27.536654 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:27.536580 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqmb6\" (UniqueName: \"kubernetes.io/projected/1059eeab-0cdf-47fe-997c-71245e8407a2-kube-api-access-nqmb6\") pod \"maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx\" (UID: \"1059eeab-0cdf-47fe-997c-71245e8407a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx" Nov 01 22:47:27.536654 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:27.536632 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1059eeab-0cdf-47fe-997c-71245e8407a2-metrics-certs\") pod \"maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx\" (UID: \"1059eeab-0cdf-47fe-997c-71245e8407a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx" Nov 01 22:47:27.536733 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:27.536690 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1059eeab-0cdf-47fe-997c-71245e8407a2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx\" (UID: \"1059eeab-0cdf-47fe-997c-71245e8407a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx" Nov 01 22:47:27.648866 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:27.640073 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1059eeab-0cdf-47fe-997c-71245e8407a2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx\" (UID: \"1059eeab-0cdf-47fe-997c-71245e8407a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx" Nov 01 22:47:27.662292 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:27.662243 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1059eeab-0cdf-47fe-997c-71245e8407a2-maas-api-tls\") pod \"maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx\" (UID: \"1059eeab-0cdf-47fe-997c-71245e8407a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx" Nov 01 22:47:27.662741 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:27.662710 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqmb6\" (UniqueName: \"kubernetes.io/projected/1059eeab-0cdf-47fe-997c-71245e8407a2-kube-api-access-nqmb6\") pod \"maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx\" (UID: \"1059eeab-0cdf-47fe-997c-71245e8407a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx" Nov 01 22:47:27.662983 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:27.662961 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1059eeab-0cdf-47fe-997c-71245e8407a2-metrics-certs\") pod \"maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx\" (UID: \"1059eeab-0cdf-47fe-997c-71245e8407a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx" Nov 01 22:47:27.663350 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:27.651714 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1059eeab-0cdf-47fe-997c-71245e8407a2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx\" (UID: \"1059eeab-0cdf-47fe-997c-71245e8407a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx" Nov 01 22:47:27.709865 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:27.699373 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1059eeab-0cdf-47fe-997c-71245e8407a2-maas-api-tls\") pod \"maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx\" (UID: \"1059eeab-0cdf-47fe-997c-71245e8407a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx" Nov 01 22:47:27.709865 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:27.709256 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqmb6\" (UniqueName: \"kubernetes.io/projected/1059eeab-0cdf-47fe-997c-71245e8407a2-kube-api-access-nqmb6\") pod \"maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx\" (UID: \"1059eeab-0cdf-47fe-997c-71245e8407a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx" Nov 01 22:47:27.725359 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:27.725320 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1059eeab-0cdf-47fe-997c-71245e8407a2-metrics-certs\") pod \"maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx\" (UID: \"1059eeab-0cdf-47fe-997c-71245e8407a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx" Nov 01 22:47:27.816120 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:27.815987 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx" Nov 01 22:47:28.281978 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:47:28.280429 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1059eeab_0cdf_47fe_997c_71245e8407a2.slice/crio-bf363baa578eef9cbb568664a10085cf092e2d1081728d1dd8de5f1a8f82d5e6 WatchSource:0}: Error finding container bf363baa578eef9cbb568664a10085cf092e2d1081728d1dd8de5f1a8f82d5e6: Status 404 returned error can't find the container with id bf363baa578eef9cbb568664a10085cf092e2d1081728d1dd8de5f1a8f82d5e6 Nov 01 22:47:28.398865 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:28.397662 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx"] Nov 01 22:47:29.367478 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:29.365465 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx" event={"ID":"1059eeab-0cdf-47fe-997c-71245e8407a2","Type":"ContainerStarted","Data":"3ceb6287b6c768d31bd814e82188b136ebabd61a5e2e14ca79de4a111348fdc8"} Nov 01 22:47:29.367478 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:29.365521 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx" event={"ID":"1059eeab-0cdf-47fe-997c-71245e8407a2","Type":"ContainerStarted","Data":"bf363baa578eef9cbb568664a10085cf092e2d1081728d1dd8de5f1a8f82d5e6"} Nov 01 22:47:29.367478 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:29.365684 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx" Nov 01 22:47:29.400885 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:29.400795 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx" podStartSLOduration=1.9034145599999999 podStartE2EDuration="2.400780173s" totalImagesPullingTime="497.365613ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:47:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:47:29.399442446 +0000 UTC m=+1417.804033079" watchObservedRunningTime="2026-11-01 22:47:29.400780173 +0000 UTC m=+1417.805370805" Nov 01 22:47:30.691973 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:30.691922 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-58c957-6bf77f96c5-lgnfs"] Nov 01 22:47:30.698387 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:30.698360 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-58c957-6bf77f96c5-lgnfs" Nov 01 22:47:30.806903 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:30.801551 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-58c957-dockercfg-h9297\"" Nov 01 22:47:30.806903 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:30.801648 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-58c957\"" Nov 01 22:47:30.808321 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:30.808282 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a09edcbb-afbe-471a-8f37-001071ed1aec-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-58c957-6bf77f96c5-lgnfs\" (UID: \"a09edcbb-afbe-471a-8f37-001071ed1aec\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-58c957-6bf77f96c5-lgnfs" Nov 01 22:47:30.813053 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:30.813014 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86zsv\" (UniqueName: \"kubernetes.io/projected/a09edcbb-afbe-471a-8f37-001071ed1aec-kube-api-access-86zsv\") pod \"payload-pre-processing-e2e-inf-a-58c957-6bf77f96c5-lgnfs\" (UID: \"a09edcbb-afbe-471a-8f37-001071ed1aec\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-58c957-6bf77f96c5-lgnfs" Nov 01 22:47:30.978955 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:30.889325 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-58c957-6bf77f96c5-lgnfs"] Nov 01 22:47:31.114846 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:31.114788 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a09edcbb-afbe-471a-8f37-001071ed1aec-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-58c957-6bf77f96c5-lgnfs\" (UID: \"a09edcbb-afbe-471a-8f37-001071ed1aec\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-58c957-6bf77f96c5-lgnfs" Nov 01 22:47:31.115043 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:31.114974 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86zsv\" (UniqueName: \"kubernetes.io/projected/a09edcbb-afbe-471a-8f37-001071ed1aec-kube-api-access-86zsv\") pod \"payload-pre-processing-e2e-inf-a-58c957-6bf77f96c5-lgnfs\" (UID: \"a09edcbb-afbe-471a-8f37-001071ed1aec\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-58c957-6bf77f96c5-lgnfs" Nov 01 22:47:31.115741 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:31.115716 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a09edcbb-afbe-471a-8f37-001071ed1aec-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-58c957-6bf77f96c5-lgnfs\" (UID: \"a09edcbb-afbe-471a-8f37-001071ed1aec\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-58c957-6bf77f96c5-lgnfs" Nov 01 22:47:31.129019 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:31.128986 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86zsv\" (UniqueName: \"kubernetes.io/projected/a09edcbb-afbe-471a-8f37-001071ed1aec-kube-api-access-86zsv\") pod \"payload-pre-processing-e2e-inf-a-58c957-6bf77f96c5-lgnfs\" (UID: \"a09edcbb-afbe-471a-8f37-001071ed1aec\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-58c957-6bf77f96c5-lgnfs" Nov 01 22:47:31.376558 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:31.376240 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-58c957-6bf77f96c5-lgnfs" Nov 01 22:47:31.773218 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:31.773191 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-58c957-6bf77f96c5-lgnfs"] Nov 01 22:47:31.774792 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:47:31.774763 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda09edcbb_afbe_471a_8f37_001071ed1aec.slice/crio-76273ec68fccf079bc2f0e3c996a1fc4fd5fb8b863a3cd1ace08dca4abd1d017 WatchSource:0}: Error finding container 76273ec68fccf079bc2f0e3c996a1fc4fd5fb8b863a3cd1ace08dca4abd1d017: Status 404 returned error can't find the container with id 76273ec68fccf079bc2f0e3c996a1fc4fd5fb8b863a3cd1ace08dca4abd1d017 Nov 01 22:47:32.455858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:32.447475 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-58c957-6bf77f96c5-lgnfs" event={"ID":"a09edcbb-afbe-471a-8f37-001071ed1aec","Type":"ContainerStarted","Data":"8d8a9af1fbb335cb6de7fbee31787f1ff030501eda1f2e941a8497b15980188f"} Nov 01 22:47:32.455858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:32.447544 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-58c957-6bf77f96c5-lgnfs" event={"ID":"a09edcbb-afbe-471a-8f37-001071ed1aec","Type":"ContainerStarted","Data":"76273ec68fccf079bc2f0e3c996a1fc4fd5fb8b863a3cd1ace08dca4abd1d017"} Nov 01 22:47:32.455858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:32.449722 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-58c957-6bf77f96c5-lgnfs" Nov 01 22:47:32.488854 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:32.485242 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-58c957-6bf77f96c5-lgnfs" podStartSLOduration=2.485221898 podStartE2EDuration="2.485221898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:47:32.484773497 +0000 UTC m=+1420.889364199" watchObservedRunningTime="2026-11-01 22:47:32.485221898 +0000 UTC m=+1420.889812531" Nov 01 22:47:34.716407 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:34.716370 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e202e7-687db7885-b9p7c"] Nov 01 22:47:34.722543 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:34.722521 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e202e7-687db7885-b9p7c" Nov 01 22:47:34.727561 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:34.727534 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e202e7-serving-cert\"" Nov 01 22:47:34.734347 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:34.734322 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e202e7-687db7885-b9p7c"] Nov 01 22:47:34.771805 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:34.771758 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41ea1680-d0bb-4926-bb24-3322cf8eeda8-maas-api-tls\") pod \"maas-api-e2e-inf-a-e202e7-687db7885-b9p7c\" (UID: \"41ea1680-d0bb-4926-bb24-3322cf8eeda8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e202e7-687db7885-b9p7c" Nov 01 22:47:34.772155 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:34.771811 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41ea1680-d0bb-4926-bb24-3322cf8eeda8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e202e7-687db7885-b9p7c\" (UID: \"41ea1680-d0bb-4926-bb24-3322cf8eeda8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e202e7-687db7885-b9p7c" Nov 01 22:47:34.775611 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:34.775580 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41ea1680-d0bb-4926-bb24-3322cf8eeda8-metrics-certs\") pod \"maas-api-e2e-inf-a-e202e7-687db7885-b9p7c\" (UID: \"41ea1680-d0bb-4926-bb24-3322cf8eeda8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e202e7-687db7885-b9p7c" Nov 01 22:47:34.775750 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:34.775676 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8scp\" (UniqueName: \"kubernetes.io/projected/41ea1680-d0bb-4926-bb24-3322cf8eeda8-kube-api-access-z8scp\") pod \"maas-api-e2e-inf-a-e202e7-687db7885-b9p7c\" (UID: \"41ea1680-d0bb-4926-bb24-3322cf8eeda8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e202e7-687db7885-b9p7c" Nov 01 22:47:34.877337 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:34.877301 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41ea1680-d0bb-4926-bb24-3322cf8eeda8-metrics-certs\") pod \"maas-api-e2e-inf-a-e202e7-687db7885-b9p7c\" (UID: \"41ea1680-d0bb-4926-bb24-3322cf8eeda8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e202e7-687db7885-b9p7c" Nov 01 22:47:34.878423 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:34.877400 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8scp\" (UniqueName: \"kubernetes.io/projected/41ea1680-d0bb-4926-bb24-3322cf8eeda8-kube-api-access-z8scp\") pod \"maas-api-e2e-inf-a-e202e7-687db7885-b9p7c\" (UID: \"41ea1680-d0bb-4926-bb24-3322cf8eeda8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e202e7-687db7885-b9p7c" Nov 01 22:47:34.878423 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:34.877510 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41ea1680-d0bb-4926-bb24-3322cf8eeda8-maas-api-tls\") pod \"maas-api-e2e-inf-a-e202e7-687db7885-b9p7c\" (UID: \"41ea1680-d0bb-4926-bb24-3322cf8eeda8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e202e7-687db7885-b9p7c" Nov 01 22:47:34.878423 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:34.877528 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41ea1680-d0bb-4926-bb24-3322cf8eeda8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e202e7-687db7885-b9p7c\" (UID: \"41ea1680-d0bb-4926-bb24-3322cf8eeda8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e202e7-687db7885-b9p7c" Nov 01 22:47:34.878423 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:47:34.877617 5011 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-e202e7-serving-cert: secret "maas-api-e2e-inf-a-e202e7-serving-cert" not found Nov 01 22:47:34.878423 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:47:34.877702 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41ea1680-d0bb-4926-bb24-3322cf8eeda8-maas-api-tls podName:41ea1680-d0bb-4926-bb24-3322cf8eeda8 nodeName:}" failed. No retries permitted until 2026-11-01 22:47:35.377681175 +0000 UTC m=+1423.782271798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/41ea1680-d0bb-4926-bb24-3322cf8eeda8-maas-api-tls") pod "maas-api-e2e-inf-a-e202e7-687db7885-b9p7c" (UID: "41ea1680-d0bb-4926-bb24-3322cf8eeda8") : secret "maas-api-e2e-inf-a-e202e7-serving-cert" not found Nov 01 22:47:34.878423 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:34.878365 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41ea1680-d0bb-4926-bb24-3322cf8eeda8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e202e7-687db7885-b9p7c\" (UID: \"41ea1680-d0bb-4926-bb24-3322cf8eeda8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e202e7-687db7885-b9p7c" Nov 01 22:47:34.880185 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:34.880129 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41ea1680-d0bb-4926-bb24-3322cf8eeda8-metrics-certs\") pod \"maas-api-e2e-inf-a-e202e7-687db7885-b9p7c\" (UID: \"41ea1680-d0bb-4926-bb24-3322cf8eeda8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e202e7-687db7885-b9p7c" Nov 01 22:47:34.892282 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:34.892257 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8scp\" (UniqueName: \"kubernetes.io/projected/41ea1680-d0bb-4926-bb24-3322cf8eeda8-kube-api-access-z8scp\") pod \"maas-api-e2e-inf-a-e202e7-687db7885-b9p7c\" (UID: \"41ea1680-d0bb-4926-bb24-3322cf8eeda8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e202e7-687db7885-b9p7c" Nov 01 22:47:35.381106 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:35.381079 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx" Nov 01 22:47:35.390733 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:35.390697 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41ea1680-d0bb-4926-bb24-3322cf8eeda8-maas-api-tls\") pod \"maas-api-e2e-inf-a-e202e7-687db7885-b9p7c\" (UID: \"41ea1680-d0bb-4926-bb24-3322cf8eeda8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e202e7-687db7885-b9p7c" Nov 01 22:47:35.393756 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:35.393725 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41ea1680-d0bb-4926-bb24-3322cf8eeda8-maas-api-tls\") pod \"maas-api-e2e-inf-a-e202e7-687db7885-b9p7c\" (UID: \"41ea1680-d0bb-4926-bb24-3322cf8eeda8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e202e7-687db7885-b9p7c" Nov 01 22:47:35.491934 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:35.491892 5011 generic.go:356] "Generic (PLEG): container finished" podID="b054e19b-487f-4cde-9752-607550ccb6c7" containerID="da35b309d0236f9fdb10c5ed70695ef90d376636ba9b32715e055500e7fab362" exitCode=0 Nov 01 22:47:35.492084 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:35.491955 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" event={"ID":"b054e19b-487f-4cde-9752-607550ccb6c7","Type":"ContainerDied","Data":"da35b309d0236f9fdb10c5ed70695ef90d376636ba9b32715e055500e7fab362"} Nov 01 22:47:35.492367 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:35.492350 5011 scope.go:122] "RemoveContainer" containerID="da35b309d0236f9fdb10c5ed70695ef90d376636ba9b32715e055500e7fab362" Nov 01 22:47:35.494100 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:35.494072 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:47:35.494168 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:35.494144 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:47:35.494204 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:35.494179 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:47:35.647841 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:35.647731 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e202e7-687db7885-b9p7c" Nov 01 22:47:35.918714 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:35.917685 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:47:36.403646 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:47:36.403615 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ea1680_d0bb_4926_bb24_3322cf8eeda8.slice/crio-73028741a178e527bb90bff6a9b233045c573b6c0205b30e6fb4e3efdb102848 WatchSource:0}: Error finding container 73028741a178e527bb90bff6a9b233045c573b6c0205b30e6fb4e3efdb102848: Status 404 returned error can't find the container with id 73028741a178e527bb90bff6a9b233045c573b6c0205b30e6fb4e3efdb102848 Nov 01 22:47:36.404135 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:36.404109 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e202e7-687db7885-b9p7c"] Nov 01 22:47:36.503110 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:36.503068 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" event={"ID":"b054e19b-487f-4cde-9752-607550ccb6c7","Type":"ContainerStarted","Data":"1f4c4cc28fc85669a438aa6efec6ab97dd5d7e5716c03df31fbb355a3d889c08"} Nov 01 22:47:36.505546 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:36.505520 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e202e7-687db7885-b9p7c" event={"ID":"41ea1680-d0bb-4926-bb24-3322cf8eeda8","Type":"ContainerStarted","Data":"73028741a178e527bb90bff6a9b233045c573b6c0205b30e6fb4e3efdb102848"} Nov 01 22:47:36.922132 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:36.922101 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:47:36.923797 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:36.923769 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 01 22:47:36.923962 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:36.923844 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" podUID="b054e19b-487f-4cde-9752-607550ccb6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 01 22:47:37.515910 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:37.515865 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e202e7-687db7885-b9p7c" event={"ID":"41ea1680-d0bb-4926-bb24-3322cf8eeda8","Type":"ContainerStarted","Data":"dba42f2a24dde556181d17b6f4b21a768859841a1d0d3ab53a88f800b311450a"} Nov 01 22:47:37.516095 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:37.515966 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e202e7-687db7885-b9p7c" Nov 01 22:47:37.536117 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:37.536066 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e202e7-687db7885-b9p7c" podStartSLOduration=3.142412653 podStartE2EDuration="3.536050017s" totalImagesPullingTime="393.637364ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:47:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:47:37.534711186 +0000 UTC m=+1425.939301820" watchObservedRunningTime="2026-11-01 22:47:37.536050017 +0000 UTC m=+1425.940640651" Nov 01 22:47:37.925863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:37.921960 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 01 22:47:37.925863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:37.922034 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" podUID="b054e19b-487f-4cde-9752-607550ccb6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 01 22:47:38.921537 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:38.921505 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 01 22:47:38.921712 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:38.921556 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" podUID="b054e19b-487f-4cde-9752-607550ccb6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 01 22:47:39.921994 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:39.921962 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 01 22:47:39.922380 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:39.922017 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" podUID="b054e19b-487f-4cde-9752-607550ccb6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 01 22:47:40.922191 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:40.922157 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 01 22:47:40.922644 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:40.922207 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" podUID="b054e19b-487f-4cde-9752-607550ccb6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 01 22:47:41.921671 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:41.921634 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 01 22:47:41.921902 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:41.921870 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" podUID="b054e19b-487f-4cde-9752-607550ccb6c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 01 22:47:42.937911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:42.937882 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:47:42.938709 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:42.938612 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:47:42.940049 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:42.940027 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 22:47:43.544670 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:43.544642 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e202e7-687db7885-b9p7c" Nov 01 22:47:44.485738 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:47:44.485708 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-58c957-6bf77f96c5-lgnfs" Nov 01 22:47:46.961222 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:47:46.961187 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:48:04.774534 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:04.774423 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:48:09.237339 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:09.237306 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f07902-558cf9ffcf-nxkzp"] Nov 01 22:48:09.359985 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:09.359949 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f07902-558cf9ffcf-nxkzp" Nov 01 22:48:09.387910 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:09.387868 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmntf\" (UniqueName: \"kubernetes.io/projected/01447e5b-5c6c-48b1-a455-765af224343f-kube-api-access-cmntf\") pod \"payload-pre-processing-e2e-inf-b-f07902-558cf9ffcf-nxkzp\" (UID: \"01447e5b-5c6c-48b1-a455-765af224343f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f07902-558cf9ffcf-nxkzp" Nov 01 22:48:09.743526 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:09.414235 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01447e5b-5c6c-48b1-a455-765af224343f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f07902-558cf9ffcf-nxkzp\" (UID: \"01447e5b-5c6c-48b1-a455-765af224343f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f07902-558cf9ffcf-nxkzp" Nov 01 22:48:09.861692 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:09.510169 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f07902\"" Nov 01 22:48:09.872095 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:09.510386 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f07902-dockercfg-tfrnt\"" Nov 01 22:48:10.165129 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:10.159614 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01447e5b-5c6c-48b1-a455-765af224343f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f07902-558cf9ffcf-nxkzp\" (UID: \"01447e5b-5c6c-48b1-a455-765af224343f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f07902-558cf9ffcf-nxkzp" Nov 01 22:48:10.165129 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:10.160976 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01447e5b-5c6c-48b1-a455-765af224343f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f07902-558cf9ffcf-nxkzp\" (UID: \"01447e5b-5c6c-48b1-a455-765af224343f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f07902-558cf9ffcf-nxkzp" Nov 01 22:48:10.165129 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:10.161042 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmntf\" (UniqueName: \"kubernetes.io/projected/01447e5b-5c6c-48b1-a455-765af224343f-kube-api-access-cmntf\") pod \"payload-pre-processing-e2e-inf-b-f07902-558cf9ffcf-nxkzp\" (UID: \"01447e5b-5c6c-48b1-a455-765af224343f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f07902-558cf9ffcf-nxkzp" Nov 01 22:48:10.227101 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:10.227068 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmntf\" (UniqueName: \"kubernetes.io/projected/01447e5b-5c6c-48b1-a455-765af224343f-kube-api-access-cmntf\") pod \"payload-pre-processing-e2e-inf-b-f07902-558cf9ffcf-nxkzp\" (UID: \"01447e5b-5c6c-48b1-a455-765af224343f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f07902-558cf9ffcf-nxkzp" Nov 01 22:48:10.420940 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:10.419505 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f07902-558cf9ffcf-nxkzp"] Nov 01 22:48:10.462656 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:10.462620 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f07902-558cf9ffcf-nxkzp" Nov 01 22:48:10.827807 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:10.827438 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f07902-558cf9ffcf-nxkzp"] Nov 01 22:48:11.407882 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:11.407365 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f07902-558cf9ffcf-nxkzp" event={"ID":"01447e5b-5c6c-48b1-a455-765af224343f","Type":"ContainerStarted","Data":"1c253276b4586b6b4d42b5633d9cf55921ce6389b9864df8025860fef89b837f"} Nov 01 22:48:11.407882 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:11.407433 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f07902-558cf9ffcf-nxkzp" event={"ID":"01447e5b-5c6c-48b1-a455-765af224343f","Type":"ContainerStarted","Data":"b7cb153666b58cda1f64d6bf80f9731395178b98e262213417c452d5064705d6"} Nov 01 22:48:11.407882 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:11.407470 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f07902-558cf9ffcf-nxkzp" Nov 01 22:48:11.448650 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:11.448593 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f07902-558cf9ffcf-nxkzp" podStartSLOduration=2.448577029 podStartE2EDuration="2.448577029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:48:11.447163594 +0000 UTC m=+1459.851754228" watchObservedRunningTime="2026-11-01 22:48:11.448577029 +0000 UTC m=+1459.853167717" Nov 01 22:48:16.963114 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:48:16.963080 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:48:22.419939 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:22.419909 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f07902-558cf9ffcf-nxkzp" Nov 01 22:48:26.717029 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:26.716971 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0e716d-cbd5484dc-t8bxj"] Nov 01 22:48:26.726883 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:26.724191 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e716d-cbd5484dc-t8bxj" Nov 01 22:48:26.729119 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:26.729095 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-0e716d\"" Nov 01 22:48:26.729521 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:26.729503 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-0e716d-dockercfg-pqtqh\"" Nov 01 22:48:26.732705 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:26.732682 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0e716d-cbd5484dc-t8bxj"] Nov 01 22:48:26.776264 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:26.776216 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk4kr\" (UniqueName: \"kubernetes.io/projected/8d0fc024-7f80-4ca6-a234-3210cee311e4-kube-api-access-fk4kr\") pod \"payload-pre-processing-e2e-inf-b-0e716d-cbd5484dc-t8bxj\" (UID: \"8d0fc024-7f80-4ca6-a234-3210cee311e4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e716d-cbd5484dc-t8bxj" Nov 01 22:48:26.776436 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:26.776292 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d0fc024-7f80-4ca6-a234-3210cee311e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0e716d-cbd5484dc-t8bxj\" (UID: \"8d0fc024-7f80-4ca6-a234-3210cee311e4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e716d-cbd5484dc-t8bxj" Nov 01 22:48:26.907857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:26.907288 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk4kr\" (UniqueName: \"kubernetes.io/projected/8d0fc024-7f80-4ca6-a234-3210cee311e4-kube-api-access-fk4kr\") pod \"payload-pre-processing-e2e-inf-b-0e716d-cbd5484dc-t8bxj\" (UID: \"8d0fc024-7f80-4ca6-a234-3210cee311e4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e716d-cbd5484dc-t8bxj" Nov 01 22:48:26.907857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:26.907369 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d0fc024-7f80-4ca6-a234-3210cee311e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0e716d-cbd5484dc-t8bxj\" (UID: \"8d0fc024-7f80-4ca6-a234-3210cee311e4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e716d-cbd5484dc-t8bxj" Nov 01 22:48:26.944236 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:26.944199 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d0fc024-7f80-4ca6-a234-3210cee311e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0e716d-cbd5484dc-t8bxj\" (UID: \"8d0fc024-7f80-4ca6-a234-3210cee311e4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e716d-cbd5484dc-t8bxj" Nov 01 22:48:26.973428 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:26.973344 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk4kr\" (UniqueName: \"kubernetes.io/projected/8d0fc024-7f80-4ca6-a234-3210cee311e4-kube-api-access-fk4kr\") pod \"payload-pre-processing-e2e-inf-b-0e716d-cbd5484dc-t8bxj\" (UID: \"8d0fc024-7f80-4ca6-a234-3210cee311e4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e716d-cbd5484dc-t8bxj" Nov 01 22:48:27.047944 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:27.047609 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e716d-cbd5484dc-t8bxj" Nov 01 22:48:27.500698 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:48:27.500658 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d0fc024_7f80_4ca6_a234_3210cee311e4.slice/crio-f9e0d406fbd1a5e524ccda42b729120c6f810c9f8c3cee5bd205ff96cd7b00d9 WatchSource:0}: Error finding container f9e0d406fbd1a5e524ccda42b729120c6f810c9f8c3cee5bd205ff96cd7b00d9: Status 404 returned error can't find the container with id f9e0d406fbd1a5e524ccda42b729120c6f810c9f8c3cee5bd205ff96cd7b00d9 Nov 01 22:48:27.504860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:27.503687 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0e716d-cbd5484dc-t8bxj"] Nov 01 22:48:28.069118 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:28.069081 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e716d-cbd5484dc-t8bxj" event={"ID":"8d0fc024-7f80-4ca6-a234-3210cee311e4","Type":"ContainerStarted","Data":"8071f887dcd95d13880e6dfcfe45248caf33924a778f5fdab44680ce9ed00825"} Nov 01 22:48:28.069603 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:28.069167 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e716d-cbd5484dc-t8bxj" Nov 01 22:48:28.069603 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:28.069187 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e716d-cbd5484dc-t8bxj" event={"ID":"8d0fc024-7f80-4ca6-a234-3210cee311e4","Type":"ContainerStarted","Data":"f9e0d406fbd1a5e524ccda42b729120c6f810c9f8c3cee5bd205ff96cd7b00d9"} Nov 01 22:48:28.092122 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:28.092069 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e716d-cbd5484dc-t8bxj" podStartSLOduration=2.092055768 podStartE2EDuration="2.092055768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:48:28.08863111 +0000 UTC m=+1476.493221929" watchObservedRunningTime="2026-11-01 22:48:28.092055768 +0000 UTC m=+1476.496646400" Nov 01 22:48:32.832877 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:32.832820 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt"] Nov 01 22:48:32.851766 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:32.851731 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt"] Nov 01 22:48:32.852007 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:32.851874 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt" Nov 01 22:48:32.860191 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:32.860165 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f07902-serving-cert\"" Nov 01 22:48:32.863964 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:32.863936 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6708ae18-425b-46be-87c4-2b5023e9ba8e-metrics-certs\") pod \"maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt\" (UID: \"6708ae18-425b-46be-87c4-2b5023e9ba8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt" Nov 01 22:48:32.864399 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:32.864112 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6708ae18-425b-46be-87c4-2b5023e9ba8e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt\" (UID: \"6708ae18-425b-46be-87c4-2b5023e9ba8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt" Nov 01 22:48:32.864399 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:32.864223 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7drb\" (UniqueName: \"kubernetes.io/projected/6708ae18-425b-46be-87c4-2b5023e9ba8e-kube-api-access-t7drb\") pod \"maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt\" (UID: \"6708ae18-425b-46be-87c4-2b5023e9ba8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt" Nov 01 22:48:32.864399 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:32.864287 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6708ae18-425b-46be-87c4-2b5023e9ba8e-maas-api-tls\") pod \"maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt\" (UID: \"6708ae18-425b-46be-87c4-2b5023e9ba8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt" Nov 01 22:48:32.965283 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:32.965250 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6708ae18-425b-46be-87c4-2b5023e9ba8e-metrics-certs\") pod \"maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt\" (UID: \"6708ae18-425b-46be-87c4-2b5023e9ba8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt" Nov 01 22:48:32.965457 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:32.965425 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6708ae18-425b-46be-87c4-2b5023e9ba8e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt\" (UID: \"6708ae18-425b-46be-87c4-2b5023e9ba8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt" Nov 01 22:48:32.965514 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:32.965504 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7drb\" (UniqueName: \"kubernetes.io/projected/6708ae18-425b-46be-87c4-2b5023e9ba8e-kube-api-access-t7drb\") pod \"maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt\" (UID: \"6708ae18-425b-46be-87c4-2b5023e9ba8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt" Nov 01 22:48:32.965606 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:32.965591 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6708ae18-425b-46be-87c4-2b5023e9ba8e-maas-api-tls\") pod \"maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt\" (UID: \"6708ae18-425b-46be-87c4-2b5023e9ba8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt" Nov 01 22:48:32.966116 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:32.966087 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6708ae18-425b-46be-87c4-2b5023e9ba8e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt\" (UID: \"6708ae18-425b-46be-87c4-2b5023e9ba8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt" Nov 01 22:48:32.968055 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:32.968033 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6708ae18-425b-46be-87c4-2b5023e9ba8e-maas-api-tls\") pod \"maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt\" (UID: \"6708ae18-425b-46be-87c4-2b5023e9ba8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt" Nov 01 22:48:32.968140 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:32.968055 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6708ae18-425b-46be-87c4-2b5023e9ba8e-metrics-certs\") pod \"maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt\" (UID: \"6708ae18-425b-46be-87c4-2b5023e9ba8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt" Nov 01 22:48:32.979381 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:32.979358 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7drb\" (UniqueName: \"kubernetes.io/projected/6708ae18-425b-46be-87c4-2b5023e9ba8e-kube-api-access-t7drb\") pod \"maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt\" (UID: \"6708ae18-425b-46be-87c4-2b5023e9ba8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt" Nov 01 22:48:33.179019 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:33.178986 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt" Nov 01 22:48:33.565862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:33.564294 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt"] Nov 01 22:48:33.614873 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:48:33.610960 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6708ae18_425b_46be_87c4_2b5023e9ba8e.slice/crio-4b7cecd7fd3720fb48ac4c35a5e74501d14908d14cc28c3851626bf95569a547 WatchSource:0}: Error finding container 4b7cecd7fd3720fb48ac4c35a5e74501d14908d14cc28c3851626bf95569a547: Status 404 returned error can't find the container with id 4b7cecd7fd3720fb48ac4c35a5e74501d14908d14cc28c3851626bf95569a547 Nov 01 22:48:34.185025 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:34.184991 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt" event={"ID":"6708ae18-425b-46be-87c4-2b5023e9ba8e","Type":"ContainerStarted","Data":"4b7cecd7fd3720fb48ac4c35a5e74501d14908d14cc28c3851626bf95569a547"} Nov 01 22:48:35.196123 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:35.196083 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt" event={"ID":"6708ae18-425b-46be-87c4-2b5023e9ba8e","Type":"ContainerStarted","Data":"62f1fce8c878fe46c1b4dad9765070fc3d0e0dc3395ddf7b3c891b62f728914c"} Nov 01 22:48:35.196659 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:35.196189 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt" Nov 01 22:48:35.223291 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:35.223235 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt" podStartSLOduration=2.6587347279999998 podStartE2EDuration="3.223215463s" totalImagesPullingTime="564.480735ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:48:35.216551616 +0000 UTC m=+1483.621142250" watchObservedRunningTime="2026-11-01 22:48:35.223215463 +0000 UTC m=+1483.627806098" Nov 01 22:48:39.110513 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:39.110483 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e716d-cbd5484dc-t8bxj" Nov 01 22:48:41.243631 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:41.243590 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt" Nov 01 22:48:41.480404 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:41.480360 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b34688-798c5d88d5-vk4cj"] Nov 01 22:48:41.495864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:41.495760 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b34688-69c5567c75-sdw9t"] Nov 01 22:48:46.135664 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:46.135625 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn"] Nov 01 22:48:46.142094 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:46.142067 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn" Nov 01 22:48:46.152747 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:46.152710 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-0e716d-serving-cert\"" Nov 01 22:48:46.161041 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:46.161013 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn"] Nov 01 22:48:46.253630 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:46.253593 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a061f44-9252-4c10-9677-912c562c47ad-metrics-certs\") pod \"maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn\" (UID: \"0a061f44-9252-4c10-9677-912c562c47ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn" Nov 01 22:48:46.253873 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:46.253771 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mlvn\" (UniqueName: \"kubernetes.io/projected/0a061f44-9252-4c10-9677-912c562c47ad-kube-api-access-4mlvn\") pod \"maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn\" (UID: \"0a061f44-9252-4c10-9677-912c562c47ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn" Nov 01 22:48:46.253873 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:46.253807 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a061f44-9252-4c10-9677-912c562c47ad-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn\" (UID: \"0a061f44-9252-4c10-9677-912c562c47ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn" Nov 01 22:48:46.254120 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:46.254092 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a061f44-9252-4c10-9677-912c562c47ad-maas-api-tls\") pod \"maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn\" (UID: \"0a061f44-9252-4c10-9677-912c562c47ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn" Nov 01 22:48:46.364746 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:46.364709 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mlvn\" (UniqueName: \"kubernetes.io/projected/0a061f44-9252-4c10-9677-912c562c47ad-kube-api-access-4mlvn\") pod \"maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn\" (UID: \"0a061f44-9252-4c10-9677-912c562c47ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn" Nov 01 22:48:46.365013 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:46.364770 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a061f44-9252-4c10-9677-912c562c47ad-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn\" (UID: \"0a061f44-9252-4c10-9677-912c562c47ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn" Nov 01 22:48:46.365108 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:46.365076 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a061f44-9252-4c10-9677-912c562c47ad-maas-api-tls\") pod \"maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn\" (UID: \"0a061f44-9252-4c10-9677-912c562c47ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn" Nov 01 22:48:46.365210 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:46.365189 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a061f44-9252-4c10-9677-912c562c47ad-metrics-certs\") pod \"maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn\" (UID: \"0a061f44-9252-4c10-9677-912c562c47ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn" Nov 01 22:48:46.366336 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:46.366307 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a061f44-9252-4c10-9677-912c562c47ad-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn\" (UID: \"0a061f44-9252-4c10-9677-912c562c47ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn" Nov 01 22:48:46.366662 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:48:46.366151 5011 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-serving-cert: secret "maas-api-e2e-inf-b-0e716d-serving-cert" not found Nov 01 22:48:46.377745 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:48:46.366914 5011 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a061f44-9252-4c10-9677-912c562c47ad-maas-api-tls podName:0a061f44-9252-4c10-9677-912c562c47ad nodeName:}" failed. No retries permitted until 2026-11-01 22:48:46.866891224 +0000 UTC m=+1495.271481849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0a061f44-9252-4c10-9677-912c562c47ad-maas-api-tls") pod "maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn" (UID: "0a061f44-9252-4c10-9677-912c562c47ad") : secret "maas-api-e2e-inf-b-0e716d-serving-cert" not found Nov 01 22:48:46.411896 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:46.408611 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a061f44-9252-4c10-9677-912c562c47ad-metrics-certs\") pod \"maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn\" (UID: \"0a061f44-9252-4c10-9677-912c562c47ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn" Nov 01 22:48:46.411896 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:46.411223 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mlvn\" (UniqueName: \"kubernetes.io/projected/0a061f44-9252-4c10-9677-912c562c47ad-kube-api-access-4mlvn\") pod \"maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn\" (UID: \"0a061f44-9252-4c10-9677-912c562c47ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn" Nov 01 22:48:46.481004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:46.480938 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-b34688-798c5d88d5-vk4cj" podUID="e2273f99-763d-41a5-bcdd-af0cf9626a2d" containerName="payload-processing" containerID="cri-o://de6691806d9e1505344e0d057f2203d1bf4a0155b522e3532b54372279bc88b3" gracePeriod=25 Nov 01 22:48:46.496586 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:46.496544 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34688-69c5567c75-sdw9t" podUID="8e43bcf5-a6f3-4119-af1b-45fb66056b1a" containerName="payload-pre-processing" containerID="cri-o://da1b9d015c7fc48a4d5dc5e94521cb616d12d6660d2b29fcdc31c9ba35913a0d" gracePeriod=25 Nov 01 22:48:46.616646 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:46.616612 5011 generic.go:356] "Generic (PLEG): container finished" podID="e2273f99-763d-41a5-bcdd-af0cf9626a2d" containerID="de6691806d9e1505344e0d057f2203d1bf4a0155b522e3532b54372279bc88b3" exitCode=0 Nov 01 22:48:46.616766 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:46.616675 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b34688-798c5d88d5-vk4cj" event={"ID":"e2273f99-763d-41a5-bcdd-af0cf9626a2d","Type":"ContainerDied","Data":"de6691806d9e1505344e0d057f2203d1bf4a0155b522e3532b54372279bc88b3"} Nov 01 22:48:46.812161 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:46.812133 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b34688-798c5d88d5-vk4cj" Nov 01 22:48:46.838168 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:46.838135 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34688-69c5567c75-sdw9t" Nov 01 22:48:46.885857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:46.883642 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2273f99-763d-41a5-bcdd-af0cf9626a2d-kube-api-access-9wxzn\" (UniqueName: \"kubernetes.io/projected/e2273f99-763d-41a5-bcdd-af0cf9626a2d-kube-api-access-9wxzn\") pod \"e2273f99-763d-41a5-bcdd-af0cf9626a2d\" (UID: \"e2273f99-763d-41a5-bcdd-af0cf9626a2d\") " Nov 01 22:48:46.885857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:46.883707 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2273f99-763d-41a5-bcdd-af0cf9626a2d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2273f99-763d-41a5-bcdd-af0cf9626a2d-plugins-config-volume\") pod \"e2273f99-763d-41a5-bcdd-af0cf9626a2d\" (UID: \"e2273f99-763d-41a5-bcdd-af0cf9626a2d\") " Nov 01 22:48:46.885857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:46.884667 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a061f44-9252-4c10-9677-912c562c47ad-maas-api-tls\") pod \"maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn\" (UID: \"0a061f44-9252-4c10-9677-912c562c47ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn" Nov 01 22:48:46.986918 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:46.981176 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2273f99-763d-41a5-bcdd-af0cf9626a2d-plugins-config-volume" pod "e2273f99-763d-41a5-bcdd-af0cf9626a2d" (UID: "e2273f99-763d-41a5-bcdd-af0cf9626a2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:48:47.085929 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:47.085883 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2273f99-763d-41a5-bcdd-af0cf9626a2d-kube-api-access-9wxzn" pod "e2273f99-763d-41a5-bcdd-af0cf9626a2d" (UID: "e2273f99-763d-41a5-bcdd-af0cf9626a2d"). InnerVolumeSpecName "kube-api-access-9wxzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:48:47.105748 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:47.099398 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e43bcf5-a6f3-4119-af1b-45fb66056b1a-kube-api-access-x4ldf\" (UniqueName: \"kubernetes.io/projected/8e43bcf5-a6f3-4119-af1b-45fb66056b1a-kube-api-access-x4ldf\") pod \"8e43bcf5-a6f3-4119-af1b-45fb66056b1a\" (UID: \"8e43bcf5-a6f3-4119-af1b-45fb66056b1a\") " Nov 01 22:48:47.105748 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:47.099637 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e43bcf5-a6f3-4119-af1b-45fb66056b1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e43bcf5-a6f3-4119-af1b-45fb66056b1a-plugins-config-volume\") pod \"8e43bcf5-a6f3-4119-af1b-45fb66056b1a\" (UID: \"8e43bcf5-a6f3-4119-af1b-45fb66056b1a\") " Nov 01 22:48:47.110480 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:47.110436 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e43bcf5-a6f3-4119-af1b-45fb66056b1a-plugins-config-volume" pod "8e43bcf5-a6f3-4119-af1b-45fb66056b1a" (UID: "8e43bcf5-a6f3-4119-af1b-45fb66056b1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:48:47.113987 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:47.113932 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e43bcf5-a6f3-4119-af1b-45fb66056b1a-kube-api-access-x4ldf" pod "8e43bcf5-a6f3-4119-af1b-45fb66056b1a" (UID: "8e43bcf5-a6f3-4119-af1b-45fb66056b1a"). InnerVolumeSpecName "kube-api-access-x4ldf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:48:47.124892 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:47.122652 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a061f44-9252-4c10-9677-912c562c47ad-maas-api-tls\") pod \"maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn\" (UID: \"0a061f44-9252-4c10-9677-912c562c47ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn" Nov 01 22:48:47.124892 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:47.123715 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4ldf\" (UniqueName: \"kubernetes.io/projected/8e43bcf5-a6f3-4119-af1b-45fb66056b1a-kube-api-access-x4ldf\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:47.124892 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:47.123744 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e43bcf5-a6f3-4119-af1b-45fb66056b1a-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:47.124892 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:47.123759 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wxzn\" (UniqueName: \"kubernetes.io/projected/e2273f99-763d-41a5-bcdd-af0cf9626a2d-kube-api-access-9wxzn\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:47.124892 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:47.123773 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2273f99-763d-41a5-bcdd-af0cf9626a2d-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:47.177395 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:48:47.177350 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:48:47.370116 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:47.370028 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn" Nov 01 22:48:47.637912 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:47.636870 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b34688-798c5d88d5-vk4cj" Nov 01 22:48:47.637912 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:47.636912 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b34688-798c5d88d5-vk4cj" event={"ID":"e2273f99-763d-41a5-bcdd-af0cf9626a2d","Type":"ContainerDied","Data":"1d61d5b6e2fc807b3921d63a5fee622bd5eddf715c2ce1c44915ec8ea8fde7ea"} Nov 01 22:48:47.637912 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:47.637023 5011 scope.go:122] "RemoveContainer" containerID="de6691806d9e1505344e0d057f2203d1bf4a0155b522e3532b54372279bc88b3" Nov 01 22:48:47.648725 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:47.643000 5011 generic.go:356] "Generic (PLEG): container finished" podID="8e43bcf5-a6f3-4119-af1b-45fb66056b1a" containerID="da1b9d015c7fc48a4d5dc5e94521cb616d12d6660d2b29fcdc31c9ba35913a0d" exitCode=0 Nov 01 22:48:47.648725 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:47.643067 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34688-69c5567c75-sdw9t" event={"ID":"8e43bcf5-a6f3-4119-af1b-45fb66056b1a","Type":"ContainerDied","Data":"da1b9d015c7fc48a4d5dc5e94521cb616d12d6660d2b29fcdc31c9ba35913a0d"} Nov 01 22:48:47.648725 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:47.643118 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34688-69c5567c75-sdw9t" event={"ID":"8e43bcf5-a6f3-4119-af1b-45fb66056b1a","Type":"ContainerDied","Data":"d86a32282a34099692d25cec4efce8cfc17fde714f52ad93a69124e5be0ee4dc"} Nov 01 22:48:47.648725 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:47.643230 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34688-69c5567c75-sdw9t" Nov 01 22:48:47.665175 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:47.665079 5011 scope.go:122] "RemoveContainer" containerID="da1b9d015c7fc48a4d5dc5e94521cb616d12d6660d2b29fcdc31c9ba35913a0d" Nov 01 22:48:47.674340 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:47.674311 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b34688-798c5d88d5-vk4cj"] Nov 01 22:48:47.680003 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:47.679981 5011 scope.go:122] "RemoveContainer" containerID="da1b9d015c7fc48a4d5dc5e94521cb616d12d6660d2b29fcdc31c9ba35913a0d" Nov 01 22:48:47.680379 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:48:47.680351 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da1b9d015c7fc48a4d5dc5e94521cb616d12d6660d2b29fcdc31c9ba35913a0d\": container with ID starting with da1b9d015c7fc48a4d5dc5e94521cb616d12d6660d2b29fcdc31c9ba35913a0d not found: ID does not exist" containerID="da1b9d015c7fc48a4d5dc5e94521cb616d12d6660d2b29fcdc31c9ba35913a0d" Nov 01 22:48:47.680480 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:47.680389 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da1b9d015c7fc48a4d5dc5e94521cb616d12d6660d2b29fcdc31c9ba35913a0d"} err="failed to get container status \"da1b9d015c7fc48a4d5dc5e94521cb616d12d6660d2b29fcdc31c9ba35913a0d\": rpc error: code = NotFound desc = could not find container \"da1b9d015c7fc48a4d5dc5e94521cb616d12d6660d2b29fcdc31c9ba35913a0d\": container with ID starting with da1b9d015c7fc48a4d5dc5e94521cb616d12d6660d2b29fcdc31c9ba35913a0d not found: ID does not exist" Nov 01 22:48:47.681873 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:47.681844 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b34688-798c5d88d5-vk4cj"] Nov 01 22:48:47.689879 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:47.689856 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b34688-69c5567c75-sdw9t"] Nov 01 22:48:47.694556 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:47.694529 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b34688-69c5567c75-sdw9t"] Nov 01 22:48:47.704047 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:48:47.704022 5011 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2273f99_763d_41a5_bcdd_af0cf9626a2d.slice/crio-1d61d5b6e2fc807b3921d63a5fee622bd5eddf715c2ce1c44915ec8ea8fde7ea\": RecentStats: unable to find data in memory cache]" Nov 01 22:48:47.749937 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:47.749905 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e43bcf5-a6f3-4119-af1b-45fb66056b1a" path="/var/lib/kubelet/pods/8e43bcf5-a6f3-4119-af1b-45fb66056b1a/volumes" Nov 01 22:48:47.750374 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:47.750358 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2273f99-763d-41a5-bcdd-af0cf9626a2d" path="/var/lib/kubelet/pods/e2273f99-763d-41a5-bcdd-af0cf9626a2d/volumes" Nov 01 22:48:47.771053 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:47.771025 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn"] Nov 01 22:48:47.771225 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:48:47.771205 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a061f44_9252_4c10_9677_912c562c47ad.slice/crio-72a04f9c0fdf8c77acdb1d84f029e42fa5549dee2c09fed88849f3f56f201fc6 WatchSource:0}: Error finding container 72a04f9c0fdf8c77acdb1d84f029e42fa5549dee2c09fed88849f3f56f201fc6: Status 404 returned error can't find the container with id 72a04f9c0fdf8c77acdb1d84f029e42fa5549dee2c09fed88849f3f56f201fc6 Nov 01 22:48:48.655248 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:48.655169 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn" event={"ID":"0a061f44-9252-4c10-9677-912c562c47ad","Type":"ContainerStarted","Data":"00e8386df3c8f82ff3b2a2ee9fc295de50367f9657fcba9aa53cbc8d146ecc38"} Nov 01 22:48:48.655248 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:48.655220 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn" event={"ID":"0a061f44-9252-4c10-9677-912c562c47ad","Type":"ContainerStarted","Data":"72a04f9c0fdf8c77acdb1d84f029e42fa5549dee2c09fed88849f3f56f201fc6"} Nov 01 22:48:48.655678 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:48.655262 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn" Nov 01 22:48:48.678160 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:48.678108 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn" podStartSLOduration=2.250695702 podStartE2EDuration="2.678093615s" totalImagesPullingTime="427.397913ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:48:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:48:48.675688619 +0000 UTC m=+1497.080279253" watchObservedRunningTime="2026-11-01 22:48:48.678093615 +0000 UTC m=+1497.082684246" Nov 01 22:48:50.759089 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:50.759020 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6"] Nov 01 22:48:50.762598 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:50.762556 5011 cpu_manager.go:414] "containerMap: removing container" podUID="8e43bcf5-a6f3-4119-af1b-45fb66056b1a" containerName="payload-pre-processing" Nov 01 22:48:50.762820 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:50.762802 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e43bcf5-a6f3-4119-af1b-45fb66056b1a" containerName="payload-pre-processing" Nov 01 22:48:50.762995 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:50.762980 5011 cpu_manager.go:414] "containerMap: removing container" podUID="e2273f99-763d-41a5-bcdd-af0cf9626a2d" containerName="payload-processing" Nov 01 22:48:50.763098 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:50.763085 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2273f99-763d-41a5-bcdd-af0cf9626a2d" containerName="payload-processing" Nov 01 22:48:50.768850 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:50.768810 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2273f99-763d-41a5-bcdd-af0cf9626a2d" containerName="payload-processing" Nov 01 22:48:50.769096 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:50.769081 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e43bcf5-a6f3-4119-af1b-45fb66056b1a" containerName="payload-pre-processing" Nov 01 22:48:50.784456 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:50.784421 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:48:50.786128 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:50.786092 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6"] Nov 01 22:48:50.791700 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:50.791377 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e36490-openshift-default-dockercfg-vjrhb\"" Nov 01 22:48:50.845303 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:50.845253 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-workload-socket\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:48:50.845810 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:50.845781 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-credential-socket\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:48:50.846099 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:50.846078 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-istio-envoy\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:48:50.846471 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:50.846450 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-workload-certs\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:48:50.949948 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:50.949902 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-workload-socket\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:48:50.950132 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:50.949962 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3927dac-a03b-43fd-bdd4-6959001771f1-istio-token\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:48:50.950309 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:50.950280 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-credential-socket\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:48:50.950456 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:50.950309 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-workload-socket\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:48:50.950456 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:50.950327 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-istio-envoy\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:48:50.950642 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:50.950624 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-workload-certs\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:48:50.950759 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:50.950740 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-istio-data\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:48:50.950848 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:50.950776 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3927dac-a03b-43fd-bdd4-6959001771f1-istio-podinfo\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:48:50.950848 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:50.950789 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-credential-socket\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:48:50.950848 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:50.950815 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqf8j\" (UniqueName: \"kubernetes.io/projected/b3927dac-a03b-43fd-bdd4-6959001771f1-kube-api-access-bqf8j\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:48:50.951010 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:50.950935 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-workload-certs\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:48:50.951069 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:50.951016 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3927dac-a03b-43fd-bdd4-6959001771f1-istiod-ca-cert\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:48:50.952742 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:50.952720 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-istio-envoy\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:48:51.052969 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:51.052877 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3927dac-a03b-43fd-bdd4-6959001771f1-istio-token\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:48:51.053151 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:51.053061 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-istio-data\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:48:51.053151 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:51.053081 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3927dac-a03b-43fd-bdd4-6959001771f1-istio-podinfo\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:48:51.053151 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:51.053103 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqf8j\" (UniqueName: \"kubernetes.io/projected/b3927dac-a03b-43fd-bdd4-6959001771f1-kube-api-access-bqf8j\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:48:51.053325 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:51.053165 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3927dac-a03b-43fd-bdd4-6959001771f1-istiod-ca-cert\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:48:51.053440 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:51.053416 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-istio-data\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:48:51.053964 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:51.053936 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3927dac-a03b-43fd-bdd4-6959001771f1-istiod-ca-cert\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:48:51.055738 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:51.055706 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3927dac-a03b-43fd-bdd4-6959001771f1-istio-podinfo\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:48:51.061113 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:51.061091 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3927dac-a03b-43fd-bdd4-6959001771f1-istio-token\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:48:51.061242 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:51.061217 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqf8j\" (UniqueName: \"kubernetes.io/projected/b3927dac-a03b-43fd-bdd4-6959001771f1-kube-api-access-bqf8j\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:48:51.137225 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:51.137189 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:48:51.443189 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:48:51.443158 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3927dac_a03b_43fd_bdd4_6959001771f1.slice/crio-e345e16628feb28eaa9396d62f52eea3a274da0742416df1acb8903fc34b3334 WatchSource:0}: Error finding container e345e16628feb28eaa9396d62f52eea3a274da0742416df1acb8903fc34b3334: Status 404 returned error can't find the container with id e345e16628feb28eaa9396d62f52eea3a274da0742416df1acb8903fc34b3334 Nov 01 22:48:51.443354 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:51.443330 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6"] Nov 01 22:48:51.446395 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:51.446363 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:48:51.446487 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:51.446433 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:48:51.446487 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:51.446460 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:48:51.683516 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:51.682675 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" event={"ID":"b3927dac-a03b-43fd-bdd4-6959001771f1","Type":"ContainerStarted","Data":"61efa79764cbd2db12dd90c1d828467c85fdbdaefa1dba34e75cc0dc4e53e4db"} Nov 01 22:48:51.683516 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:51.682737 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" event={"ID":"b3927dac-a03b-43fd-bdd4-6959001771f1","Type":"ContainerStarted","Data":"e345e16628feb28eaa9396d62f52eea3a274da0742416df1acb8903fc34b3334"} Nov 01 22:48:51.704262 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:51.704147 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" podStartSLOduration=1.704130186 podStartE2EDuration="1.704130186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:48:51.702572833 +0000 UTC m=+1500.107163489" watchObservedRunningTime="2026-11-01 22:48:51.704130186 +0000 UTC m=+1500.108720817" Nov 01 22:48:52.137480 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:52.137386 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:48:52.139440 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:52.139407 5011 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 01 22:48:52.139591 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:52.139473 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" podUID="b3927dac-a03b-43fd-bdd4-6959001771f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 01 22:48:53.137730 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:53.137693 5011 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 01 22:48:53.138157 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:53.137753 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" podUID="b3927dac-a03b-43fd-bdd4-6959001771f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 01 22:48:54.168040 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:54.167937 5011 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 01 22:48:54.168040 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:54.167999 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" podUID="b3927dac-a03b-43fd-bdd4-6959001771f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 01 22:48:54.674057 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:54.674026 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn" Nov 01 22:48:55.143575 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:55.143490 5011 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 01 22:48:55.143813 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:55.143786 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" podUID="b3927dac-a03b-43fd-bdd4-6959001771f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 01 22:48:56.137777 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:56.137744 5011 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 01 22:48:56.138310 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:56.137800 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" podUID="b3927dac-a03b-43fd-bdd4-6959001771f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 01 22:48:57.137764 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:57.137730 5011 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 01 22:48:57.137965 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:57.137785 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" podUID="b3927dac-a03b-43fd-bdd4-6959001771f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 01 22:48:58.138573 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:58.138534 5011 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 01 22:48:58.139181 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:58.138616 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" podUID="b3927dac-a03b-43fd-bdd4-6959001771f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 01 22:48:59.145144 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:59.145108 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:48:59.145647 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:59.145606 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:48:59.146348 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:48:59.146324 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:49:03.726441 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:03.726399 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-021714-7b74898bd7-s2gqb"] Nov 01 22:49:03.733702 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:03.733503 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-021714-7b74898bd7-s2gqb" Nov 01 22:49:03.748043 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:03.747957 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-021714-dockercfg-qjjv6\"" Nov 01 22:49:03.758716 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:03.758686 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-021714-7b74898bd7-s2gqb"] Nov 01 22:49:03.799959 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:03.799926 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-021714\"" Nov 01 22:49:03.954858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:03.951108 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m42g\" (UniqueName: \"kubernetes.io/projected/02975ed3-18c1-49ee-8bae-e86edb4e0130-kube-api-access-2m42g\") pod \"payload-processing-e2e-inf-a-021714-7b74898bd7-s2gqb\" (UID: \"02975ed3-18c1-49ee-8bae-e86edb4e0130\") " pod="openshift-ingress/payload-processing-e2e-inf-a-021714-7b74898bd7-s2gqb" Nov 01 22:49:03.983937 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:03.983847 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02975ed3-18c1-49ee-8bae-e86edb4e0130-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-021714-7b74898bd7-s2gqb\" (UID: \"02975ed3-18c1-49ee-8bae-e86edb4e0130\") " pod="openshift-ingress/payload-processing-e2e-inf-a-021714-7b74898bd7-s2gqb" Nov 01 22:49:04.141926 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:04.130075 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m42g\" (UniqueName: \"kubernetes.io/projected/02975ed3-18c1-49ee-8bae-e86edb4e0130-kube-api-access-2m42g\") pod \"payload-processing-e2e-inf-a-021714-7b74898bd7-s2gqb\" (UID: \"02975ed3-18c1-49ee-8bae-e86edb4e0130\") " pod="openshift-ingress/payload-processing-e2e-inf-a-021714-7b74898bd7-s2gqb" Nov 01 22:49:04.142241 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:04.142216 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02975ed3-18c1-49ee-8bae-e86edb4e0130-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-021714-7b74898bd7-s2gqb\" (UID: \"02975ed3-18c1-49ee-8bae-e86edb4e0130\") " pod="openshift-ingress/payload-processing-e2e-inf-a-021714-7b74898bd7-s2gqb" Nov 01 22:49:04.190872 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:04.143043 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02975ed3-18c1-49ee-8bae-e86edb4e0130-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-021714-7b74898bd7-s2gqb\" (UID: \"02975ed3-18c1-49ee-8bae-e86edb4e0130\") " pod="openshift-ingress/payload-processing-e2e-inf-a-021714-7b74898bd7-s2gqb" Nov 01 22:49:04.190872 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:04.174852 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m42g\" (UniqueName: \"kubernetes.io/projected/02975ed3-18c1-49ee-8bae-e86edb4e0130-kube-api-access-2m42g\") pod \"payload-processing-e2e-inf-a-021714-7b74898bd7-s2gqb\" (UID: \"02975ed3-18c1-49ee-8bae-e86edb4e0130\") " pod="openshift-ingress/payload-processing-e2e-inf-a-021714-7b74898bd7-s2gqb" Nov 01 22:49:04.359780 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:04.359692 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-021714-7b74898bd7-s2gqb" Nov 01 22:49:04.674631 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:49:04.674601 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02975ed3_18c1_49ee_8bae_e86edb4e0130.slice/crio-c2f48fa1c1327f802a24a8327e0d401ade358c66f857d9936997d29124c434df WatchSource:0}: Error finding container c2f48fa1c1327f802a24a8327e0d401ade358c66f857d9936997d29124c434df: Status 404 returned error can't find the container with id c2f48fa1c1327f802a24a8327e0d401ade358c66f857d9936997d29124c434df Nov 01 22:49:04.675224 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:04.675199 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-021714-7b74898bd7-s2gqb"] Nov 01 22:49:05.096896 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:05.096776 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-021714-7b74898bd7-s2gqb" event={"ID":"02975ed3-18c1-49ee-8bae-e86edb4e0130","Type":"ContainerStarted","Data":"fe589a25553a50e5ab341d7a39a63745bab2a6c58e2d8ea925ffad304e292bcf"} Nov 01 22:49:05.096896 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:05.096861 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-021714-7b74898bd7-s2gqb" event={"ID":"02975ed3-18c1-49ee-8bae-e86edb4e0130","Type":"ContainerStarted","Data":"c2f48fa1c1327f802a24a8327e0d401ade358c66f857d9936997d29124c434df"} Nov 01 22:49:05.097341 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:05.096980 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-021714-7b74898bd7-s2gqb" Nov 01 22:49:05.103321 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:05.103282 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c0d076-9775f9644-dn545"] Nov 01 22:49:05.103549 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:05.103520 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c0d076-9775f9644-dn545" podUID="51471588-b877-4ffe-ba04-0e80882dfcd6" containerName="maas-api" containerID="cri-o://68e5febefcf341b606a34caf78faffb45e743a40bfa496018fe87a24f4f41092" gracePeriod=30 Nov 01 22:49:05.166133 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:05.166073 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-021714-7b74898bd7-s2gqb" podStartSLOduration=2.166051605 podStartE2EDuration="2.166051605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:05.134105882 +0000 UTC m=+1513.538696551" watchObservedRunningTime="2026-11-01 22:49:05.166051605 +0000 UTC m=+1513.570642239" Nov 01 22:49:05.762379 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:05.762303 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c0d076-9775f9644-dn545" Nov 01 22:49:05.872989 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:05.872901 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51471588-b877-4ffe-ba04-0e80882dfcd6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/51471588-b877-4ffe-ba04-0e80882dfcd6-metrics-certs\") pod \"51471588-b877-4ffe-ba04-0e80882dfcd6\" (UID: \"51471588-b877-4ffe-ba04-0e80882dfcd6\") " Nov 01 22:49:05.873189 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:05.873171 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51471588-b877-4ffe-ba04-0e80882dfcd6-kube-api-access-jb5m8\" (UniqueName: \"kubernetes.io/projected/51471588-b877-4ffe-ba04-0e80882dfcd6-kube-api-access-jb5m8\") pod \"51471588-b877-4ffe-ba04-0e80882dfcd6\" (UID: \"51471588-b877-4ffe-ba04-0e80882dfcd6\") " Nov 01 22:49:05.873248 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:05.873212 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51471588-b877-4ffe-ba04-0e80882dfcd6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51471588-b877-4ffe-ba04-0e80882dfcd6-openshift-service-ca\") pod \"51471588-b877-4ffe-ba04-0e80882dfcd6\" (UID: \"51471588-b877-4ffe-ba04-0e80882dfcd6\") " Nov 01 22:49:05.873296 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:05.873250 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51471588-b877-4ffe-ba04-0e80882dfcd6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51471588-b877-4ffe-ba04-0e80882dfcd6-maas-api-tls\") pod \"51471588-b877-4ffe-ba04-0e80882dfcd6\" (UID: \"51471588-b877-4ffe-ba04-0e80882dfcd6\") " Nov 01 22:49:05.873949 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:05.873921 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51471588-b877-4ffe-ba04-0e80882dfcd6-openshift-service-ca" pod "51471588-b877-4ffe-ba04-0e80882dfcd6" (UID: "51471588-b877-4ffe-ba04-0e80882dfcd6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:49:05.874080 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:05.874049 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51471588-b877-4ffe-ba04-0e80882dfcd6-openshift-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:05.875309 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:05.875282 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51471588-b877-4ffe-ba04-0e80882dfcd6-metrics-certs" pod "51471588-b877-4ffe-ba04-0e80882dfcd6" (UID: "51471588-b877-4ffe-ba04-0e80882dfcd6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:49:05.875968 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:05.875944 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51471588-b877-4ffe-ba04-0e80882dfcd6-maas-api-tls" pod "51471588-b877-4ffe-ba04-0e80882dfcd6" (UID: "51471588-b877-4ffe-ba04-0e80882dfcd6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:49:05.876041 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:05.876003 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51471588-b877-4ffe-ba04-0e80882dfcd6-kube-api-access-jb5m8" pod "51471588-b877-4ffe-ba04-0e80882dfcd6" (UID: "51471588-b877-4ffe-ba04-0e80882dfcd6"). InnerVolumeSpecName "kube-api-access-jb5m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:05.982925 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:05.980937 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51471588-b877-4ffe-ba04-0e80882dfcd6-metrics-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:05.982925 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:05.980981 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb5m8\" (UniqueName: \"kubernetes.io/projected/51471588-b877-4ffe-ba04-0e80882dfcd6-kube-api-access-jb5m8\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:05.982925 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:05.980998 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51471588-b877-4ffe-ba04-0e80882dfcd6-maas-api-tls\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:06.105622 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:06.105588 5011 generic.go:356] "Generic (PLEG): container finished" podID="51471588-b877-4ffe-ba04-0e80882dfcd6" containerID="68e5febefcf341b606a34caf78faffb45e743a40bfa496018fe87a24f4f41092" exitCode=0 Nov 01 22:49:06.106069 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:06.105635 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c0d076-9775f9644-dn545" Nov 01 22:49:06.106069 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:06.105666 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c0d076-9775f9644-dn545" event={"ID":"51471588-b877-4ffe-ba04-0e80882dfcd6","Type":"ContainerDied","Data":"68e5febefcf341b606a34caf78faffb45e743a40bfa496018fe87a24f4f41092"} Nov 01 22:49:06.106069 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:06.105725 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c0d076-9775f9644-dn545" event={"ID":"51471588-b877-4ffe-ba04-0e80882dfcd6","Type":"ContainerDied","Data":"cbdafd767840c7ef5c9606a553b8b37f88e924aa1771229981ce342a0b60c7d3"} Nov 01 22:49:06.106069 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:06.105754 5011 scope.go:122] "RemoveContainer" containerID="68e5febefcf341b606a34caf78faffb45e743a40bfa496018fe87a24f4f41092" Nov 01 22:49:06.118024 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:06.118003 5011 scope.go:122] "RemoveContainer" containerID="68e5febefcf341b606a34caf78faffb45e743a40bfa496018fe87a24f4f41092" Nov 01 22:49:06.118311 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:49:06.118292 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68e5febefcf341b606a34caf78faffb45e743a40bfa496018fe87a24f4f41092\": container with ID starting with 68e5febefcf341b606a34caf78faffb45e743a40bfa496018fe87a24f4f41092 not found: ID does not exist" containerID="68e5febefcf341b606a34caf78faffb45e743a40bfa496018fe87a24f4f41092" Nov 01 22:49:06.118373 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:06.118319 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68e5febefcf341b606a34caf78faffb45e743a40bfa496018fe87a24f4f41092"} err="failed to get container status \"68e5febefcf341b606a34caf78faffb45e743a40bfa496018fe87a24f4f41092\": rpc error: code = NotFound desc = could not find container \"68e5febefcf341b606a34caf78faffb45e743a40bfa496018fe87a24f4f41092\": container with ID starting with 68e5febefcf341b606a34caf78faffb45e743a40bfa496018fe87a24f4f41092 not found: ID does not exist" Nov 01 22:49:06.134812 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:06.134747 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c0d076-9775f9644-dn545"] Nov 01 22:49:06.139021 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:06.138996 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c0d076-9775f9644-dn545"] Nov 01 22:49:06.722915 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:06.722819 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr"] Nov 01 22:49:06.723744 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:06.723721 5011 cpu_manager.go:414] "containerMap: removing container" podUID="51471588-b877-4ffe-ba04-0e80882dfcd6" containerName="maas-api" Nov 01 22:49:06.723744 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:06.723741 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51471588-b877-4ffe-ba04-0e80882dfcd6" containerName="maas-api" Nov 01 22:49:06.723975 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:06.723926 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="51471588-b877-4ffe-ba04-0e80882dfcd6" containerName="maas-api" Nov 01 22:49:06.737874 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:06.729310 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr" Nov 01 22:49:06.738484 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:06.738461 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr"] Nov 01 22:49:06.773664 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:06.773635 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-6b78bd-serving-cert\"" Nov 01 22:49:06.792072 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:06.792030 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39d06520-7ba9-44c0-a35d-22cfce8aaf08-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr\" (UID: \"39d06520-7ba9-44c0-a35d-22cfce8aaf08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr" Nov 01 22:49:06.792312 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:06.792263 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39d06520-7ba9-44c0-a35d-22cfce8aaf08-metrics-certs\") pod \"maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr\" (UID: \"39d06520-7ba9-44c0-a35d-22cfce8aaf08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr" Nov 01 22:49:06.792387 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:06.792313 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39d06520-7ba9-44c0-a35d-22cfce8aaf08-maas-api-tls\") pod \"maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr\" (UID: \"39d06520-7ba9-44c0-a35d-22cfce8aaf08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr" Nov 01 22:49:06.792436 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:06.792395 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-542wn\" (UniqueName: \"kubernetes.io/projected/39d06520-7ba9-44c0-a35d-22cfce8aaf08-kube-api-access-542wn\") pod \"maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr\" (UID: \"39d06520-7ba9-44c0-a35d-22cfce8aaf08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr" Nov 01 22:49:06.894495 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:06.894456 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39d06520-7ba9-44c0-a35d-22cfce8aaf08-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr\" (UID: \"39d06520-7ba9-44c0-a35d-22cfce8aaf08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr" Nov 01 22:49:06.894738 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:06.894711 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39d06520-7ba9-44c0-a35d-22cfce8aaf08-metrics-certs\") pod \"maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr\" (UID: \"39d06520-7ba9-44c0-a35d-22cfce8aaf08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr" Nov 01 22:49:06.894805 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:06.894759 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39d06520-7ba9-44c0-a35d-22cfce8aaf08-maas-api-tls\") pod \"maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr\" (UID: \"39d06520-7ba9-44c0-a35d-22cfce8aaf08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr" Nov 01 22:49:06.894898 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:06.894871 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-542wn\" (UniqueName: \"kubernetes.io/projected/39d06520-7ba9-44c0-a35d-22cfce8aaf08-kube-api-access-542wn\") pod \"maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr\" (UID: \"39d06520-7ba9-44c0-a35d-22cfce8aaf08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr" Nov 01 22:49:06.895276 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:06.895255 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39d06520-7ba9-44c0-a35d-22cfce8aaf08-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr\" (UID: \"39d06520-7ba9-44c0-a35d-22cfce8aaf08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr" Nov 01 22:49:06.897553 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:06.897522 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39d06520-7ba9-44c0-a35d-22cfce8aaf08-metrics-certs\") pod \"maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr\" (UID: \"39d06520-7ba9-44c0-a35d-22cfce8aaf08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr" Nov 01 22:49:06.897701 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:06.897679 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39d06520-7ba9-44c0-a35d-22cfce8aaf08-maas-api-tls\") pod \"maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr\" (UID: \"39d06520-7ba9-44c0-a35d-22cfce8aaf08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr" Nov 01 22:49:06.907610 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:06.907588 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-542wn\" (UniqueName: \"kubernetes.io/projected/39d06520-7ba9-44c0-a35d-22cfce8aaf08-kube-api-access-542wn\") pod \"maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr\" (UID: \"39d06520-7ba9-44c0-a35d-22cfce8aaf08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr" Nov 01 22:49:07.056959 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:07.054151 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr" Nov 01 22:49:07.651288 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:07.651249 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr"] Nov 01 22:49:07.652982 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:49:07.652933 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d06520_7ba9_44c0_a35d_22cfce8aaf08.slice/crio-e094192df98e175aa9ee2922163fdd8b1a04ceaee6bef9ae9362956d25fab98d WatchSource:0}: Error finding container e094192df98e175aa9ee2922163fdd8b1a04ceaee6bef9ae9362956d25fab98d: Status 404 returned error can't find the container with id e094192df98e175aa9ee2922163fdd8b1a04ceaee6bef9ae9362956d25fab98d Nov 01 22:49:07.759232 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:07.759141 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51471588-b877-4ffe-ba04-0e80882dfcd6" path="/var/lib/kubelet/pods/51471588-b877-4ffe-ba04-0e80882dfcd6/volumes" Nov 01 22:49:07.759671 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:07.759646 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b34688-7b88d575b8-qb72d"] Nov 01 22:49:07.759934 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:07.759898 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34688-7b88d575b8-qb72d" podUID="bd9c2fec-dd43-44cc-9a0e-86cf68e5c915" containerName="maas-api" containerID="cri-o://fdfd3e100d969dad629e3740d2b8c680c29c4fff29de1269140feef25822cfa5" gracePeriod=30 Nov 01 22:49:08.312578 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:08.312349 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr" event={"ID":"39d06520-7ba9-44c0-a35d-22cfce8aaf08","Type":"ContainerStarted","Data":"e094192df98e175aa9ee2922163fdd8b1a04ceaee6bef9ae9362956d25fab98d"} Nov 01 22:49:08.315384 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:08.315279 5011 generic.go:356] "Generic (PLEG): container finished" podID="bd9c2fec-dd43-44cc-9a0e-86cf68e5c915" containerID="fdfd3e100d969dad629e3740d2b8c680c29c4fff29de1269140feef25822cfa5" exitCode=0 Nov 01 22:49:08.315962 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:08.315938 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34688-7b88d575b8-qb72d" event={"ID":"bd9c2fec-dd43-44cc-9a0e-86cf68e5c915","Type":"ContainerDied","Data":"fdfd3e100d969dad629e3740d2b8c680c29c4fff29de1269140feef25822cfa5"} Nov 01 22:49:08.408217 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:08.408163 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34688-7b88d575b8-qb72d" Nov 01 22:49:08.445234 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:08.445182 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd9c2fec-dd43-44cc-9a0e-86cf68e5c915-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd9c2fec-dd43-44cc-9a0e-86cf68e5c915-maas-api-tls\") pod \"bd9c2fec-dd43-44cc-9a0e-86cf68e5c915\" (UID: \"bd9c2fec-dd43-44cc-9a0e-86cf68e5c915\") " Nov 01 22:49:08.445376 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:08.445304 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd9c2fec-dd43-44cc-9a0e-86cf68e5c915-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd9c2fec-dd43-44cc-9a0e-86cf68e5c915-metrics-certs\") pod \"bd9c2fec-dd43-44cc-9a0e-86cf68e5c915\" (UID: \"bd9c2fec-dd43-44cc-9a0e-86cf68e5c915\") " Nov 01 22:49:08.445453 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:08.445429 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd9c2fec-dd43-44cc-9a0e-86cf68e5c915-kube-api-access-8tsmg\" (UniqueName: \"kubernetes.io/projected/bd9c2fec-dd43-44cc-9a0e-86cf68e5c915-kube-api-access-8tsmg\") pod \"bd9c2fec-dd43-44cc-9a0e-86cf68e5c915\" (UID: \"bd9c2fec-dd43-44cc-9a0e-86cf68e5c915\") " Nov 01 22:49:08.445576 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:08.445559 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd9c2fec-dd43-44cc-9a0e-86cf68e5c915-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd9c2fec-dd43-44cc-9a0e-86cf68e5c915-openshift-service-ca\") pod \"bd9c2fec-dd43-44cc-9a0e-86cf68e5c915\" (UID: \"bd9c2fec-dd43-44cc-9a0e-86cf68e5c915\") " Nov 01 22:49:08.446065 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:08.446013 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd9c2fec-dd43-44cc-9a0e-86cf68e5c915-openshift-service-ca" pod "bd9c2fec-dd43-44cc-9a0e-86cf68e5c915" (UID: "bd9c2fec-dd43-44cc-9a0e-86cf68e5c915"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:49:08.446869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:08.446847 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd9c2fec-dd43-44cc-9a0e-86cf68e5c915-openshift-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:08.448982 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:08.448955 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd9c2fec-dd43-44cc-9a0e-86cf68e5c915-metrics-certs" pod "bd9c2fec-dd43-44cc-9a0e-86cf68e5c915" (UID: "bd9c2fec-dd43-44cc-9a0e-86cf68e5c915"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:49:08.449089 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:08.448999 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd9c2fec-dd43-44cc-9a0e-86cf68e5c915-maas-api-tls" pod "bd9c2fec-dd43-44cc-9a0e-86cf68e5c915" (UID: "bd9c2fec-dd43-44cc-9a0e-86cf68e5c915"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:49:08.449089 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:08.449011 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd9c2fec-dd43-44cc-9a0e-86cf68e5c915-kube-api-access-8tsmg" pod "bd9c2fec-dd43-44cc-9a0e-86cf68e5c915" (UID: "bd9c2fec-dd43-44cc-9a0e-86cf68e5c915"). InnerVolumeSpecName "kube-api-access-8tsmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:08.548188 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:08.548153 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd9c2fec-dd43-44cc-9a0e-86cf68e5c915-maas-api-tls\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:08.548188 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:08.548179 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd9c2fec-dd43-44cc-9a0e-86cf68e5c915-metrics-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:08.548188 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:08.548189 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tsmg\" (UniqueName: \"kubernetes.io/projected/bd9c2fec-dd43-44cc-9a0e-86cf68e5c915-kube-api-access-8tsmg\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:09.325418 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:09.325386 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34688-7b88d575b8-qb72d" Nov 01 22:49:09.325885 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:09.325410 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34688-7b88d575b8-qb72d" event={"ID":"bd9c2fec-dd43-44cc-9a0e-86cf68e5c915","Type":"ContainerDied","Data":"d72ed78b5989343083aad0d05067c080bdcf885dd12b4aa2a69729a3f1052117"} Nov 01 22:49:09.325885 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:09.325493 5011 scope.go:122] "RemoveContainer" containerID="fdfd3e100d969dad629e3740d2b8c680c29c4fff29de1269140feef25822cfa5" Nov 01 22:49:09.327294 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:09.327261 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr" event={"ID":"39d06520-7ba9-44c0-a35d-22cfce8aaf08","Type":"ContainerStarted","Data":"b8ac5cc4ebe6f774ea9658e32cc36197dd8fbf9b7883b18f83a516c611a2afb9"} Nov 01 22:49:09.327427 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:09.327410 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr" Nov 01 22:49:09.367824 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:09.367755 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr" podStartSLOduration=2.580233382 podStartE2EDuration="3.367736343s" totalImagesPullingTime="787.502961ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:09.362307525 +0000 UTC m=+1517.766898159" watchObservedRunningTime="2026-11-01 22:49:09.367736343 +0000 UTC m=+1517.772326976" Nov 01 22:49:09.395957 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:09.395911 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b34688-7b88d575b8-qb72d"] Nov 01 22:49:09.407363 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:09.407333 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b34688-7b88d575b8-qb72d"] Nov 01 22:49:09.783865 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:09.783810 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd9c2fec-dd43-44cc-9a0e-86cf68e5c915" path="/var/lib/kubelet/pods/bd9c2fec-dd43-44cc-9a0e-86cf68e5c915/volumes" Nov 01 22:49:14.718406 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:14.718364 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh"] Nov 01 22:49:14.719811 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:14.719786 5011 cpu_manager.go:414] "containerMap: removing container" podUID="bd9c2fec-dd43-44cc-9a0e-86cf68e5c915" containerName="maas-api" Nov 01 22:49:14.720014 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:14.719816 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd9c2fec-dd43-44cc-9a0e-86cf68e5c915" containerName="maas-api" Nov 01 22:49:14.720115 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:14.720097 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd9c2fec-dd43-44cc-9a0e-86cf68e5c915" containerName="maas-api" Nov 01 22:49:14.736123 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:14.736078 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh"] Nov 01 22:49:14.736371 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:14.736203 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:14.769078 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:14.768303 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2f675d-openshift-default-dockercfg-hv7v4\"" Nov 01 22:49:14.952856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:14.947257 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-istio-envoy\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:14.952856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:14.947420 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-credential-socket\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:14.952856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:14.947452 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-workload-certs\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:14.952856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:14.947517 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-workload-socket\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:15.067664 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.067567 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb96044-ee0a-4050-8076-ebce2c1d990a-istiod-ca-cert\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:15.068344 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.068303 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-istio-envoy\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:15.068738 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.068707 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb96044-ee0a-4050-8076-ebce2c1d990a-istio-token\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:15.068915 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.068896 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-credential-socket\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:15.069060 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.069042 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb96044-ee0a-4050-8076-ebce2c1d990a-istio-podinfo\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:15.075522 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.075480 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-workload-certs\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:15.075800 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.075781 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-workload-socket\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:15.102887 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.102352 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-credential-socket\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:15.103103 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.103010 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-workload-certs\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:15.103671 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.103649 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4bm7\" (UniqueName: \"kubernetes.io/projected/deb96044-ee0a-4050-8076-ebce2c1d990a-kube-api-access-r4bm7\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:15.103933 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.103912 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-istio-data\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:15.104568 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.104551 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-workload-socket\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:15.149856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.148249 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-istio-envoy\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:15.240957 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.240225 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb96044-ee0a-4050-8076-ebce2c1d990a-istio-token\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:15.240957 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.240280 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb96044-ee0a-4050-8076-ebce2c1d990a-istio-podinfo\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:15.240957 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.240414 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4bm7\" (UniqueName: \"kubernetes.io/projected/deb96044-ee0a-4050-8076-ebce2c1d990a-kube-api-access-r4bm7\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:15.240957 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.240449 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-istio-data\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:15.240957 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.240574 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb96044-ee0a-4050-8076-ebce2c1d990a-istiod-ca-cert\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:15.246938 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.246526 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-istio-data\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:15.260366 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.260330 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb96044-ee0a-4050-8076-ebce2c1d990a-istiod-ca-cert\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:15.302908 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.294945 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb96044-ee0a-4050-8076-ebce2c1d990a-istio-podinfo\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:15.302908 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.296197 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb96044-ee0a-4050-8076-ebce2c1d990a-istio-token\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:15.326911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.319973 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4bm7\" (UniqueName: \"kubernetes.io/projected/deb96044-ee0a-4050-8076-ebce2c1d990a-kube-api-access-r4bm7\") pod \"e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:15.406876 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.406688 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr" Nov 01 22:49:15.441783 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.441752 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:15.662714 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.661705 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz"] Nov 01 22:49:15.670969 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.669145 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:15.679898 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.678965 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1799f8-openshift-default-dockercfg-fvhdd\"" Nov 01 22:49:15.695232 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.691777 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz"] Nov 01 22:49:15.720089 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.720057 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istio-token\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:15.720500 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.720121 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-workload-socket\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:15.720500 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.720235 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-credential-socket\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:15.720500 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.720262 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istio-podinfo\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:15.720500 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.720312 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istiod-ca-cert\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:15.720500 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.720335 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7wn7\" (UniqueName: \"kubernetes.io/projected/4923bea7-5f7a-4fb4-a064-d4ffa307379a-kube-api-access-j7wn7\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:15.720500 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.720369 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-workload-certs\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:15.720700 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.720519 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istio-data\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:15.720700 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.720613 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istio-envoy\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:15.821914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.821814 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-credential-socket\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:15.821914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.821869 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istio-podinfo\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:15.822173 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.821929 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istiod-ca-cert\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:15.822173 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.821953 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7wn7\" (UniqueName: \"kubernetes.io/projected/4923bea7-5f7a-4fb4-a064-d4ffa307379a-kube-api-access-j7wn7\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:15.822173 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.821995 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-workload-certs\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:15.822349 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.822179 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istio-data\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:15.822349 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.822266 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-credential-socket\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:15.822349 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.822329 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istio-envoy\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:15.822514 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.822364 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istio-token\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:15.822514 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.822437 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-workload-socket\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:15.822824 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.822801 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istio-data\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:15.823612 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.823587 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-workload-socket\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:15.824071 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.824050 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-workload-certs\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:15.824230 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.824207 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istiod-ca-cert\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:15.826128 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.826105 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istio-envoy\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:15.827817 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.827797 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istio-podinfo\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:15.831658 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.831628 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istio-token\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:15.831658 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.831640 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7wn7\" (UniqueName: \"kubernetes.io/projected/4923bea7-5f7a-4fb4-a064-d4ffa307379a-kube-api-access-j7wn7\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:15.876938 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.876903 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh"] Nov 01 22:49:15.877744 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:49:15.877714 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb96044_ee0a_4050_8076_ebce2c1d990a.slice/crio-8175dee3f90f2bedbda87145dd11106e07cb23938464f1b732cce8310272c5b2 WatchSource:0}: Error finding container 8175dee3f90f2bedbda87145dd11106e07cb23938464f1b732cce8310272c5b2: Status 404 returned error can't find the container with id 8175dee3f90f2bedbda87145dd11106e07cb23938464f1b732cce8310272c5b2 Nov 01 22:49:15.881345 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.881310 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:49:15.881447 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.881381 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:49:15.881447 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:15.881413 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:49:16.004929 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:16.002677 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:16.124087 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:16.124059 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-021714-7b74898bd7-s2gqb" Nov 01 22:49:16.422920 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:16.413590 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz"] Nov 01 22:49:16.434023 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:49:16.433962 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4923bea7_5f7a_4fb4_a064_d4ffa307379a.slice/crio-5125224a6a3da5c61eae4cc6a6c8f3d47c83d4a619482748badae4958192146c WatchSource:0}: Error finding container 5125224a6a3da5c61eae4cc6a6c8f3d47c83d4a619482748badae4958192146c: Status 404 returned error can't find the container with id 5125224a6a3da5c61eae4cc6a6c8f3d47c83d4a619482748badae4958192146c Nov 01 22:49:16.438282 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:16.438247 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:49:16.438459 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:16.438442 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:49:16.438552 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:16.438537 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:49:16.455544 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:16.455513 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" event={"ID":"4923bea7-5f7a-4fb4-a064-d4ffa307379a","Type":"ContainerStarted","Data":"5125224a6a3da5c61eae4cc6a6c8f3d47c83d4a619482748badae4958192146c"} Nov 01 22:49:16.457521 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:16.457498 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" event={"ID":"deb96044-ee0a-4050-8076-ebce2c1d990a","Type":"ContainerStarted","Data":"f4077be34fc1ad4c8666239c81e6092a00dba3cac0dd365251f732426419cc97"} Nov 01 22:49:16.457611 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:16.457543 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" event={"ID":"deb96044-ee0a-4050-8076-ebce2c1d990a","Type":"ContainerStarted","Data":"8175dee3f90f2bedbda87145dd11106e07cb23938464f1b732cce8310272c5b2"} Nov 01 22:49:16.495219 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:16.495152 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" podStartSLOduration=2.495131737 podStartE2EDuration="2.495131737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:16.485115298 +0000 UTC m=+1524.889705942" watchObservedRunningTime="2026-11-01 22:49:16.495131737 +0000 UTC m=+1524.899722373" Nov 01 22:49:17.276284 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:49:17.268759 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:49:17.448398 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:17.448281 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:17.457569 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:17.457521 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:17.471883 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:17.471810 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" event={"ID":"4923bea7-5f7a-4fb4-a064-d4ffa307379a","Type":"ContainerStarted","Data":"40f1dfe0d3bcc9c2b39c3bb22ab045fe55bfee32124afd322207f6d04f652bd2"} Nov 01 22:49:17.472080 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:17.472066 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:17.473154 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:17.473120 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:49:17.527654 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:17.527543 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" podStartSLOduration=2.527528222 podStartE2EDuration="2.527528222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:17.524904201 +0000 UTC m=+1525.929494834" watchObservedRunningTime="2026-11-01 22:49:17.527528222 +0000 UTC m=+1525.932118853" Nov 01 22:49:18.007412 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:18.007378 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:18.007998 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:18.007919 5011 patch_prober.go:29] interesting pod/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 01 22:49:18.008217 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:18.008185 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" podUID="4923bea7-5f7a-4fb4-a064-d4ffa307379a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 01 22:49:19.006849 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:19.005158 5011 patch_prober.go:29] interesting pod/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 01 22:49:19.006849 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:19.005217 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" podUID="4923bea7-5f7a-4fb4-a064-d4ffa307379a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 01 22:49:20.015882 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:20.003474 5011 patch_prober.go:29] interesting pod/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 01 22:49:20.015882 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:20.003551 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" podUID="4923bea7-5f7a-4fb4-a064-d4ffa307379a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 01 22:49:21.147917 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:21.147773 5011 patch_prober.go:29] interesting pod/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 01 22:49:21.147917 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:21.147863 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" podUID="4923bea7-5f7a-4fb4-a064-d4ffa307379a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 01 22:49:21.873304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:21.873263 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:22.004251 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:22.004215 5011 patch_prober.go:29] interesting pod/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 01 22:49:22.004447 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:22.004290 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" podUID="4923bea7-5f7a-4fb4-a064-d4ffa307379a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 01 22:49:23.010259 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:23.010221 5011 patch_prober.go:29] interesting pod/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 01 22:49:23.010874 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:23.010825 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" podUID="4923bea7-5f7a-4fb4-a064-d4ffa307379a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 01 22:49:24.003162 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:24.003126 5011 patch_prober.go:29] interesting pod/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Nov 01 22:49:24.003372 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:24.003190 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" podUID="4923bea7-5f7a-4fb4-a064-d4ffa307379a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Nov 01 22:49:25.009409 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:25.009376 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:25.010633 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:25.010597 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:25.011279 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:25.011256 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:49:33.581878 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:33.581777 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:49:40.106245 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:40.106166 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr"] Nov 01 22:49:40.107137 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:40.107106 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" podUID="1e83fcc9-66da-4de1-837a-b9a2a7f7a056" containerName="istio-proxy" containerID="cri-o://8c9bf38e318c286e9bc5fede40f45d98baea06953a0d1953a546e8be1306104e" gracePeriod=30 Nov 01 22:49:40.112304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:40.112276 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf"] Nov 01 22:49:40.112576 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:40.112530 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" podUID="b5730c77-aecb-441e-82f5-9d34e5e3b8d5" containerName="istio-proxy" containerID="cri-o://c377a821828cc93fe4c52895c50a58c05f2e499672010ab8b25e5285758194e0" gracePeriod=30 Nov 01 22:49:45.481330 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.481299 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:49:45.525265 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.525193 5011 generic.go:356] "Generic (PLEG): container finished" podID="1e83fcc9-66da-4de1-837a-b9a2a7f7a056" containerID="8c9bf38e318c286e9bc5fede40f45d98baea06953a0d1953a546e8be1306104e" exitCode=0 Nov 01 22:49:45.526607 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.526354 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" Nov 01 22:49:45.526607 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.526367 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:49:45.526792 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.526761 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" event={"ID":"1e83fcc9-66da-4de1-837a-b9a2a7f7a056","Type":"ContainerDied","Data":"8c9bf38e318c286e9bc5fede40f45d98baea06953a0d1953a546e8be1306104e"} Nov 01 22:49:45.527014 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.526846 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr" event={"ID":"1e83fcc9-66da-4de1-837a-b9a2a7f7a056","Type":"ContainerDied","Data":"0fb2d04d3e2725391e9be6dcbb5737eb8392f9aec1f05db5d3b918f8907e9270"} Nov 01 22:49:45.527014 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.526880 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2"] Nov 01 22:49:45.527014 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.527001 5011 scope.go:122] "RemoveContainer" containerID="8c9bf38e318c286e9bc5fede40f45d98baea06953a0d1953a546e8be1306104e" Nov 01 22:49:45.528678 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.528203 5011 cpu_manager.go:414] "containerMap: removing container" podUID="b5730c77-aecb-441e-82f5-9d34e5e3b8d5" containerName="istio-proxy" Nov 01 22:49:45.528678 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.528222 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5730c77-aecb-441e-82f5-9d34e5e3b8d5" containerName="istio-proxy" Nov 01 22:49:45.528678 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.528263 5011 cpu_manager.go:414] "containerMap: removing container" podUID="1e83fcc9-66da-4de1-837a-b9a2a7f7a056" containerName="istio-proxy" Nov 01 22:49:45.528678 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.528273 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e83fcc9-66da-4de1-837a-b9a2a7f7a056" containerName="istio-proxy" Nov 01 22:49:45.528678 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.528526 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5730c77-aecb-441e-82f5-9d34e5e3b8d5" containerName="istio-proxy" Nov 01 22:49:45.528678 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.528571 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5730c77-aecb-441e-82f5-9d34e5e3b8d5" containerName="istio-proxy" Nov 01 22:49:45.528678 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.528581 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e83fcc9-66da-4de1-837a-b9a2a7f7a056" containerName="istio-proxy" Nov 01 22:49:45.531184 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.531152 5011 generic.go:356] "Generic (PLEG): container finished" podID="b5730c77-aecb-441e-82f5-9d34e5e3b8d5" containerID="c377a821828cc93fe4c52895c50a58c05f2e499672010ab8b25e5285758194e0" exitCode=0 Nov 01 22:49:45.531386 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.531371 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" Nov 01 22:49:45.533241 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.533214 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" event={"ID":"b5730c77-aecb-441e-82f5-9d34e5e3b8d5","Type":"ContainerDied","Data":"c377a821828cc93fe4c52895c50a58c05f2e499672010ab8b25e5285758194e0"} Nov 01 22:49:45.533341 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.533262 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf" event={"ID":"b5730c77-aecb-441e-82f5-9d34e5e3b8d5","Type":"ContainerDied","Data":"61c186f25ca675982590e8641f3571e42d528da60e286875a2dd3c973abf3dc0"} Nov 01 22:49:45.533451 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.533429 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2" Nov 01 22:49:45.549277 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.545112 5011 scope.go:122] "RemoveContainer" containerID="8c9bf38e318c286e9bc5fede40f45d98baea06953a0d1953a546e8be1306104e" Nov 01 22:49:45.549277 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:49:45.545493 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c9bf38e318c286e9bc5fede40f45d98baea06953a0d1953a546e8be1306104e\": container with ID starting with 8c9bf38e318c286e9bc5fede40f45d98baea06953a0d1953a546e8be1306104e not found: ID does not exist" containerID="8c9bf38e318c286e9bc5fede40f45d98baea06953a0d1953a546e8be1306104e" Nov 01 22:49:45.549277 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.545529 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c9bf38e318c286e9bc5fede40f45d98baea06953a0d1953a546e8be1306104e"} err="failed to get container status \"8c9bf38e318c286e9bc5fede40f45d98baea06953a0d1953a546e8be1306104e\": rpc error: code = NotFound desc = could not find container \"8c9bf38e318c286e9bc5fede40f45d98baea06953a0d1953a546e8be1306104e\": container with ID starting with 8c9bf38e318c286e9bc5fede40f45d98baea06953a0d1953a546e8be1306104e not found: ID does not exist" Nov 01 22:49:45.549277 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.545551 5011 scope.go:122] "RemoveContainer" containerID="c377a821828cc93fe4c52895c50a58c05f2e499672010ab8b25e5285758194e0" Nov 01 22:49:45.549277 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.546502 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2"] Nov 01 22:49:45.555743 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.555713 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istio-token\" (UniqueName: \"kubernetes.io/projected/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istio-token\") pod \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " Nov 01 22:49:45.555906 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.555811 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-workload-certs\") pod \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " Nov 01 22:49:45.555906 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.555890 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-credential-socket\") pod \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " Nov 01 22:49:45.556055 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.555912 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-workload-socket\") pod \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " Nov 01 22:49:45.556055 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.555959 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istiod-ca-cert\") pod \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " Nov 01 22:49:45.556055 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.556003 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istio-podinfo\") pod \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " Nov 01 22:49:45.556055 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.556041 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-kube-api-access-zq9j6\" (UniqueName: \"kubernetes.io/projected/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-kube-api-access-zq9j6\") pod \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " Nov 01 22:49:45.556270 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.556110 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istio-data\") pod \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " Nov 01 22:49:45.556270 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.556160 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istio-envoy\") pod \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\" (UID: \"1e83fcc9-66da-4de1-837a-b9a2a7f7a056\") " Nov 01 22:49:45.557151 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.557125 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-workload-socket" pod "1e83fcc9-66da-4de1-837a-b9a2a7f7a056" (UID: "1e83fcc9-66da-4de1-837a-b9a2a7f7a056"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:45.557772 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.557742 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-workload-certs" pod "1e83fcc9-66da-4de1-837a-b9a2a7f7a056" (UID: "1e83fcc9-66da-4de1-837a-b9a2a7f7a056"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:45.557989 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.557967 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istio-data" pod "1e83fcc9-66da-4de1-837a-b9a2a7f7a056" (UID: "1e83fcc9-66da-4de1-837a-b9a2a7f7a056"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:45.557989 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.557976 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-credential-socket" pod "1e83fcc9-66da-4de1-837a-b9a2a7f7a056" (UID: "1e83fcc9-66da-4de1-837a-b9a2a7f7a056"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:45.558221 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.558204 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istiod-ca-cert" pod "1e83fcc9-66da-4de1-837a-b9a2a7f7a056" (UID: "1e83fcc9-66da-4de1-837a-b9a2a7f7a056"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:49:45.563156 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.563123 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-kube-api-access-zq9j6" pod "1e83fcc9-66da-4de1-837a-b9a2a7f7a056" (UID: "1e83fcc9-66da-4de1-837a-b9a2a7f7a056"). InnerVolumeSpecName "kube-api-access-zq9j6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:45.565275 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.565244 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istio-podinfo" pod "1e83fcc9-66da-4de1-837a-b9a2a7f7a056" (UID: "1e83fcc9-66da-4de1-837a-b9a2a7f7a056"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:49:45.566213 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.566185 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istio-token" pod "1e83fcc9-66da-4de1-837a-b9a2a7f7a056" (UID: "1e83fcc9-66da-4de1-837a-b9a2a7f7a056"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:45.579062 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.579027 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istio-envoy" pod "1e83fcc9-66da-4de1-837a-b9a2a7f7a056" (UID: "1e83fcc9-66da-4de1-837a-b9a2a7f7a056"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:45.579652 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.579624 5011 scope.go:122] "RemoveContainer" containerID="444caa99c6b01836eaa1d41f577683c7cf47b49f05483f5bc886609c980aab3c" Nov 01 22:49:45.605732 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.605700 5011 scope.go:122] "RemoveContainer" containerID="c377a821828cc93fe4c52895c50a58c05f2e499672010ab8b25e5285758194e0" Nov 01 22:49:45.606174 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:49:45.606145 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c377a821828cc93fe4c52895c50a58c05f2e499672010ab8b25e5285758194e0\": container with ID starting with c377a821828cc93fe4c52895c50a58c05f2e499672010ab8b25e5285758194e0 not found: ID does not exist" containerID="c377a821828cc93fe4c52895c50a58c05f2e499672010ab8b25e5285758194e0" Nov 01 22:49:45.606288 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.606187 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c377a821828cc93fe4c52895c50a58c05f2e499672010ab8b25e5285758194e0"} err="failed to get container status \"c377a821828cc93fe4c52895c50a58c05f2e499672010ab8b25e5285758194e0\": rpc error: code = NotFound desc = could not find container \"c377a821828cc93fe4c52895c50a58c05f2e499672010ab8b25e5285758194e0\": container with ID starting with c377a821828cc93fe4c52895c50a58c05f2e499672010ab8b25e5285758194e0 not found: ID does not exist" Nov 01 22:49:45.606288 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.606213 5011 scope.go:122] "RemoveContainer" containerID="444caa99c6b01836eaa1d41f577683c7cf47b49f05483f5bc886609c980aab3c" Nov 01 22:49:45.606595 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:49:45.606567 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"444caa99c6b01836eaa1d41f577683c7cf47b49f05483f5bc886609c980aab3c\": container with ID starting with 444caa99c6b01836eaa1d41f577683c7cf47b49f05483f5bc886609c980aab3c not found: ID does not exist" containerID="444caa99c6b01836eaa1d41f577683c7cf47b49f05483f5bc886609c980aab3c" Nov 01 22:49:45.606663 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.606602 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"444caa99c6b01836eaa1d41f577683c7cf47b49f05483f5bc886609c980aab3c"} err="failed to get container status \"444caa99c6b01836eaa1d41f577683c7cf47b49f05483f5bc886609c980aab3c\": rpc error: code = NotFound desc = could not find container \"444caa99c6b01836eaa1d41f577683c7cf47b49f05483f5bc886609c980aab3c\": container with ID starting with 444caa99c6b01836eaa1d41f577683c7cf47b49f05483f5bc886609c980aab3c not found: ID does not exist" Nov 01 22:49:45.608787 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.608769 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-2f675d-serving-cert\"" Nov 01 22:49:45.658236 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.658207 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istiod-ca-cert\") pod \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " Nov 01 22:49:45.658422 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.658268 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istio-envoy\") pod \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " Nov 01 22:49:45.658422 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.658368 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-workload-socket\") pod \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " Nov 01 22:49:45.658422 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.658402 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-kube-api-access-sq655\" (UniqueName: \"kubernetes.io/projected/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-kube-api-access-sq655\") pod \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " Nov 01 22:49:45.658610 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.658439 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istio-data\") pod \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " Nov 01 22:49:45.658713 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.658693 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-workload-certs\") pod \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " Nov 01 22:49:45.658817 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.658699 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-workload-socket" pod "b5730c77-aecb-441e-82f5-9d34e5e3b8d5" (UID: "b5730c77-aecb-441e-82f5-9d34e5e3b8d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:45.658817 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.658744 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istiod-ca-cert" pod "b5730c77-aecb-441e-82f5-9d34e5e3b8d5" (UID: "b5730c77-aecb-441e-82f5-9d34e5e3b8d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:49:45.658817 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.658763 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-credential-socket\") pod \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " Nov 01 22:49:45.659039 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.658847 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istio-token\" (UniqueName: \"kubernetes.io/projected/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istio-token\") pod \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " Nov 01 22:49:45.659039 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.658928 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istio-podinfo\") pod \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\" (UID: \"b5730c77-aecb-441e-82f5-9d34e5e3b8d5\") " Nov 01 22:49:45.659039 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.658992 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istio-data" pod "b5730c77-aecb-441e-82f5-9d34e5e3b8d5" (UID: "b5730c77-aecb-441e-82f5-9d34e5e3b8d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:45.659354 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.659291 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrzvp\" (UniqueName: \"kubernetes.io/projected/da68866f-9cf3-4467-b876-6ac92fcf6c3c-kube-api-access-jrzvp\") pod \"maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2\" (UID: \"da68866f-9cf3-4467-b876-6ac92fcf6c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2" Nov 01 22:49:45.659581 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.659296 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-workload-certs" pod "b5730c77-aecb-441e-82f5-9d34e5e3b8d5" (UID: "b5730c77-aecb-441e-82f5-9d34e5e3b8d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:45.659581 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.659532 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-credential-socket" pod "b5730c77-aecb-441e-82f5-9d34e5e3b8d5" (UID: "b5730c77-aecb-441e-82f5-9d34e5e3b8d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:45.660109 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.660068 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da68866f-9cf3-4467-b876-6ac92fcf6c3c-metrics-certs\") pod \"maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2\" (UID: \"da68866f-9cf3-4467-b876-6ac92fcf6c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2" Nov 01 22:49:45.660316 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.660291 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da68866f-9cf3-4467-b876-6ac92fcf6c3c-maas-api-tls\") pod \"maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2\" (UID: \"da68866f-9cf3-4467-b876-6ac92fcf6c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2" Nov 01 22:49:45.660396 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.660359 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da68866f-9cf3-4467-b876-6ac92fcf6c3c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2\" (UID: \"da68866f-9cf3-4467-b876-6ac92fcf6c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2" Nov 01 22:49:45.660694 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.660675 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:45.660777 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.660702 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:45.660777 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.660720 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:45.660777 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.660736 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:45.660777 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.660749 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:45.660777 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.660762 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:45.660777 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.660776 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:45.661083 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.660791 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:45.661083 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.660805 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:45.661083 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.660820 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq9j6\" (UniqueName: \"kubernetes.io/projected/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-kube-api-access-zq9j6\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:45.661083 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.660854 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:45.661083 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.660868 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e83fcc9-66da-4de1-837a-b9a2a7f7a056-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:45.661083 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.660884 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:45.661083 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.660899 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:45.661314 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.661090 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istio-envoy" pod "b5730c77-aecb-441e-82f5-9d34e5e3b8d5" (UID: "b5730c77-aecb-441e-82f5-9d34e5e3b8d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:45.661314 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.661194 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-kube-api-access-sq655" pod "b5730c77-aecb-441e-82f5-9d34e5e3b8d5" (UID: "b5730c77-aecb-441e-82f5-9d34e5e3b8d5"). InnerVolumeSpecName "kube-api-access-sq655". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:45.661314 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.661284 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istio-token" pod "b5730c77-aecb-441e-82f5-9d34e5e3b8d5" (UID: "b5730c77-aecb-441e-82f5-9d34e5e3b8d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:45.661560 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.661538 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istio-podinfo" pod "b5730c77-aecb-441e-82f5-9d34e5e3b8d5" (UID: "b5730c77-aecb-441e-82f5-9d34e5e3b8d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:49:45.789011 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.786328 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da68866f-9cf3-4467-b876-6ac92fcf6c3c-maas-api-tls\") pod \"maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2\" (UID: \"da68866f-9cf3-4467-b876-6ac92fcf6c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2" Nov 01 22:49:45.789011 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.786386 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da68866f-9cf3-4467-b876-6ac92fcf6c3c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2\" (UID: \"da68866f-9cf3-4467-b876-6ac92fcf6c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2" Nov 01 22:49:45.789011 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.786595 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrzvp\" (UniqueName: \"kubernetes.io/projected/da68866f-9cf3-4467-b876-6ac92fcf6c3c-kube-api-access-jrzvp\") pod \"maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2\" (UID: \"da68866f-9cf3-4467-b876-6ac92fcf6c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2" Nov 01 22:49:45.789011 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.786781 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da68866f-9cf3-4467-b876-6ac92fcf6c3c-metrics-certs\") pod \"maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2\" (UID: \"da68866f-9cf3-4467-b876-6ac92fcf6c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2" Nov 01 22:49:45.789011 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.786979 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:45.789011 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.786998 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:45.789011 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.787014 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq655\" (UniqueName: \"kubernetes.io/projected/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-kube-api-access-sq655\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:45.789011 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.787030 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5730c77-aecb-441e-82f5-9d34e5e3b8d5-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:45.789011 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.788272 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da68866f-9cf3-4467-b876-6ac92fcf6c3c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2\" (UID: \"da68866f-9cf3-4467-b876-6ac92fcf6c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2" Nov 01 22:49:45.792313 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.792280 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da68866f-9cf3-4467-b876-6ac92fcf6c3c-metrics-certs\") pod \"maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2\" (UID: \"da68866f-9cf3-4467-b876-6ac92fcf6c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2" Nov 01 22:49:45.792313 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.792287 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da68866f-9cf3-4467-b876-6ac92fcf6c3c-maas-api-tls\") pod \"maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2\" (UID: \"da68866f-9cf3-4467-b876-6ac92fcf6c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2" Nov 01 22:49:45.804974 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.804281 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrzvp\" (UniqueName: \"kubernetes.io/projected/da68866f-9cf3-4467-b876-6ac92fcf6c3c-kube-api-access-jrzvp\") pod \"maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2\" (UID: \"da68866f-9cf3-4467-b876-6ac92fcf6c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2" Nov 01 22:49:45.851029 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.850988 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr"] Nov 01 22:49:45.858235 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.858146 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2" Nov 01 22:49:45.863395 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.863345 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-klgqr"] Nov 01 22:49:45.882786 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.882759 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf"] Nov 01 22:49:45.894925 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:45.894894 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-kn6wf"] Nov 01 22:49:46.159454 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:49:46.159410 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda68866f_9cf3_4467_b876_6ac92fcf6c3c.slice/crio-a4ca22e801c151bc00e75375e950afdeee3c072a4db716e047392389cd6f3f25 WatchSource:0}: Error finding container a4ca22e801c151bc00e75375e950afdeee3c072a4db716e047392389cd6f3f25: Status 404 returned error can't find the container with id a4ca22e801c151bc00e75375e950afdeee3c072a4db716e047392389cd6f3f25 Nov 01 22:49:46.161084 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:46.161058 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2"] Nov 01 22:49:46.543352 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:46.543318 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2" event={"ID":"da68866f-9cf3-4467-b876-6ac92fcf6c3c","Type":"ContainerStarted","Data":"a4ca22e801c151bc00e75375e950afdeee3c072a4db716e047392389cd6f3f25"} Nov 01 22:49:46.968636 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:49:46.967761 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:49:47.557752 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:47.557712 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2" event={"ID":"da68866f-9cf3-4467-b876-6ac92fcf6c3c","Type":"ContainerStarted","Data":"acc33b8c62bc85baad83ad6324df1002c9ee26c9a7e80b0268d31ca81be3886b"} Nov 01 22:49:47.558317 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:47.557797 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2" Nov 01 22:49:47.579141 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:47.579077 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2" podStartSLOduration=4.11283096 podStartE2EDuration="4.579058377s" totalImagesPullingTime="466.227417ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:47.57736155 +0000 UTC m=+1555.981952185" watchObservedRunningTime="2026-11-01 22:49:47.579058377 +0000 UTC m=+1555.983649011" Nov 01 22:49:47.789151 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:47.789116 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e83fcc9-66da-4de1-837a-b9a2a7f7a056" path="/var/lib/kubelet/pods/1e83fcc9-66da-4de1-837a-b9a2a7f7a056/volumes" Nov 01 22:49:47.790322 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:47.790297 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5730c77-aecb-441e-82f5-9d34e5e3b8d5" path="/var/lib/kubelet/pods/b5730c77-aecb-441e-82f5-9d34e5e3b8d5/volumes" Nov 01 22:49:49.787512 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:49.787474 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0e685d-6f4746bcb7-l65sf"] Nov 01 22:49:49.788359 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:49.788342 5011 cpu_manager.go:414] "containerMap: removing container" podUID="b5730c77-aecb-441e-82f5-9d34e5e3b8d5" containerName="istio-proxy" Nov 01 22:49:49.788404 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:49.788360 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5730c77-aecb-441e-82f5-9d34e5e3b8d5" containerName="istio-proxy" Nov 01 22:49:49.793951 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:49.793925 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e685d-6f4746bcb7-l65sf" Nov 01 22:49:49.804714 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:49.804683 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-0e685d\"" Nov 01 22:49:49.804920 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:49.804846 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-0e685d-dockercfg-zslrc\"" Nov 01 22:49:49.811489 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:49.811449 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0e685d-6f4746bcb7-l65sf"] Nov 01 22:49:49.824824 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:49.824792 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f027946a-f04e-4d9a-babc-bc81a506c5a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0e685d-6f4746bcb7-l65sf\" (UID: \"f027946a-f04e-4d9a-babc-bc81a506c5a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e685d-6f4746bcb7-l65sf" Nov 01 22:49:49.826484 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:49.825200 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsdvd\" (UniqueName: \"kubernetes.io/projected/f027946a-f04e-4d9a-babc-bc81a506c5a4-kube-api-access-gsdvd\") pod \"payload-pre-processing-e2e-inf-b-0e685d-6f4746bcb7-l65sf\" (UID: \"f027946a-f04e-4d9a-babc-bc81a506c5a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e685d-6f4746bcb7-l65sf" Nov 01 22:49:49.835944 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:49.835921 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0e685d-784cb6758c-4nwqk"] Nov 01 22:49:49.841478 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:49.841459 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0e685d-784cb6758c-4nwqk" Nov 01 22:49:49.858640 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:49.858612 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0e685d-784cb6758c-4nwqk"] Nov 01 22:49:50.010875 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:49.988406 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b86735b-2e85-4dc0-aff1-180a64cc76d0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0e685d-784cb6758c-4nwqk\" (UID: \"8b86735b-2e85-4dc0-aff1-180a64cc76d0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0e685d-784cb6758c-4nwqk" Nov 01 22:49:50.053414 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:50.050914 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-445fp\" (UniqueName: \"kubernetes.io/projected/8b86735b-2e85-4dc0-aff1-180a64cc76d0-kube-api-access-445fp\") pod \"payload-processing-e2e-inf-b-0e685d-784cb6758c-4nwqk\" (UID: \"8b86735b-2e85-4dc0-aff1-180a64cc76d0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0e685d-784cb6758c-4nwqk" Nov 01 22:49:50.053414 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:50.050982 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f027946a-f04e-4d9a-babc-bc81a506c5a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0e685d-6f4746bcb7-l65sf\" (UID: \"f027946a-f04e-4d9a-babc-bc81a506c5a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e685d-6f4746bcb7-l65sf" Nov 01 22:49:50.066736 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:50.066703 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f027946a-f04e-4d9a-babc-bc81a506c5a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0e685d-6f4746bcb7-l65sf\" (UID: \"f027946a-f04e-4d9a-babc-bc81a506c5a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e685d-6f4746bcb7-l65sf" Nov 01 22:49:50.083192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:50.080522 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsdvd\" (UniqueName: \"kubernetes.io/projected/f027946a-f04e-4d9a-babc-bc81a506c5a4-kube-api-access-gsdvd\") pod \"payload-pre-processing-e2e-inf-b-0e685d-6f4746bcb7-l65sf\" (UID: \"f027946a-f04e-4d9a-babc-bc81a506c5a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e685d-6f4746bcb7-l65sf" Nov 01 22:49:50.276858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:50.222684 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b86735b-2e85-4dc0-aff1-180a64cc76d0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0e685d-784cb6758c-4nwqk\" (UID: \"8b86735b-2e85-4dc0-aff1-180a64cc76d0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0e685d-784cb6758c-4nwqk" Nov 01 22:49:50.354744 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:50.354647 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-445fp\" (UniqueName: \"kubernetes.io/projected/8b86735b-2e85-4dc0-aff1-180a64cc76d0-kube-api-access-445fp\") pod \"payload-processing-e2e-inf-b-0e685d-784cb6758c-4nwqk\" (UID: \"8b86735b-2e85-4dc0-aff1-180a64cc76d0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0e685d-784cb6758c-4nwqk" Nov 01 22:49:50.355219 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:50.333604 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsdvd\" (UniqueName: \"kubernetes.io/projected/f027946a-f04e-4d9a-babc-bc81a506c5a4-kube-api-access-gsdvd\") pod \"payload-pre-processing-e2e-inf-b-0e685d-6f4746bcb7-l65sf\" (UID: \"f027946a-f04e-4d9a-babc-bc81a506c5a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e685d-6f4746bcb7-l65sf" Nov 01 22:49:50.355390 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:50.279234 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b86735b-2e85-4dc0-aff1-180a64cc76d0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0e685d-784cb6758c-4nwqk\" (UID: \"8b86735b-2e85-4dc0-aff1-180a64cc76d0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0e685d-784cb6758c-4nwqk" Nov 01 22:49:50.368126 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:50.367756 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-445fp\" (UniqueName: \"kubernetes.io/projected/8b86735b-2e85-4dc0-aff1-180a64cc76d0-kube-api-access-445fp\") pod \"payload-processing-e2e-inf-b-0e685d-784cb6758c-4nwqk\" (UID: \"8b86735b-2e85-4dc0-aff1-180a64cc76d0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0e685d-784cb6758c-4nwqk" Nov 01 22:49:50.414261 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:50.414230 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e685d-6f4746bcb7-l65sf" Nov 01 22:49:50.466391 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:50.466361 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0e685d-784cb6758c-4nwqk" Nov 01 22:49:50.890317 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:50.890284 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0e685d-784cb6758c-4nwqk"] Nov 01 22:49:50.891048 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:49:50.890996 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b86735b_2e85_4dc0_aff1_180a64cc76d0.slice/crio-07c3c01a3f9b2771bc44cb5c63c46133495f347b6014d1ca54c9ad5053dcc791 WatchSource:0}: Error finding container 07c3c01a3f9b2771bc44cb5c63c46133495f347b6014d1ca54c9ad5053dcc791: Status 404 returned error can't find the container with id 07c3c01a3f9b2771bc44cb5c63c46133495f347b6014d1ca54c9ad5053dcc791 Nov 01 22:49:50.909572 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:50.909541 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0e685d-6f4746bcb7-l65sf"] Nov 01 22:49:50.909683 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:49:50.909624 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf027946a_f04e_4d9a_babc_bc81a506c5a4.slice/crio-708fd263f173a73f507cc2e596f5f31af8352594d9be8ea635126b9d517761c7 WatchSource:0}: Error finding container 708fd263f173a73f507cc2e596f5f31af8352594d9be8ea635126b9d517761c7: Status 404 returned error can't find the container with id 708fd263f173a73f507cc2e596f5f31af8352594d9be8ea635126b9d517761c7 Nov 01 22:49:51.643159 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:51.643119 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e685d-6f4746bcb7-l65sf" event={"ID":"f027946a-f04e-4d9a-babc-bc81a506c5a4","Type":"ContainerStarted","Data":"410b3c9029c1da2443fed8159242888d7373a69852645b3316787ae030ece2b7"} Nov 01 22:49:51.643362 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:51.643216 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e685d-6f4746bcb7-l65sf" event={"ID":"f027946a-f04e-4d9a-babc-bc81a506c5a4","Type":"ContainerStarted","Data":"708fd263f173a73f507cc2e596f5f31af8352594d9be8ea635126b9d517761c7"} Nov 01 22:49:51.643362 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:51.643329 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e685d-6f4746bcb7-l65sf" Nov 01 22:49:51.644888 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:51.644860 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0e685d-784cb6758c-4nwqk" event={"ID":"8b86735b-2e85-4dc0-aff1-180a64cc76d0","Type":"ContainerStarted","Data":"2b40ca1600c210fa01c93a2b9f3d74a781a2e7c2ba575bd15e3ced40059049cc"} Nov 01 22:49:51.645025 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:51.644901 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0e685d-784cb6758c-4nwqk" event={"ID":"8b86735b-2e85-4dc0-aff1-180a64cc76d0","Type":"ContainerStarted","Data":"07c3c01a3f9b2771bc44cb5c63c46133495f347b6014d1ca54c9ad5053dcc791"} Nov 01 22:49:51.645025 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:51.644977 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0e685d-784cb6758c-4nwqk" Nov 01 22:49:51.666248 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:51.666199 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e685d-6f4746bcb7-l65sf" podStartSLOduration=2.666186354 podStartE2EDuration="2.666186354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:51.662804936 +0000 UTC m=+1560.067395569" watchObservedRunningTime="2026-11-01 22:49:51.666186354 +0000 UTC m=+1560.070777020" Nov 01 22:49:51.687438 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:51.687388 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-0e685d-784cb6758c-4nwqk" podStartSLOduration=2.687373916 podStartE2EDuration="2.687373916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:51.683592733 +0000 UTC m=+1560.088183366" watchObservedRunningTime="2026-11-01 22:49:51.687373916 +0000 UTC m=+1560.091964564" Nov 01 22:49:52.771295 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:52.771263 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh"] Nov 01 22:49:52.776444 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:52.776423 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:49:52.788100 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:52.788080 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e202e7-openshift-default-dockercfg-s5jpf\"" Nov 01 22:49:52.793245 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:52.793221 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh"] Nov 01 22:49:52.893035 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:52.883133 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34d081d4-e311-40f4-9695-abcc8ff40605-istio-podinfo\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:49:52.893035 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:52.883189 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-workload-certs\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:49:52.893035 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:52.883235 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-istio-envoy\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:49:52.893035 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:52.883304 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-istio-data\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:49:52.893035 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:52.883443 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28mfh\" (UniqueName: \"kubernetes.io/projected/34d081d4-e311-40f4-9695-abcc8ff40605-kube-api-access-28mfh\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:49:52.893035 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:52.883523 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34d081d4-e311-40f4-9695-abcc8ff40605-istiod-ca-cert\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:49:52.893035 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:52.883558 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-credential-socket\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:49:52.893035 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:52.883680 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-workload-socket\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:49:52.893035 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:52.883733 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34d081d4-e311-40f4-9695-abcc8ff40605-istio-token\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:49:52.988580 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:52.988552 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34d081d4-e311-40f4-9695-abcc8ff40605-istio-podinfo\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:49:52.988945 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:52.988919 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-workload-certs\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:49:52.989041 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:52.988984 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-istio-envoy\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:49:52.989095 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:52.989056 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-istio-data\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:49:52.989189 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:52.989172 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28mfh\" (UniqueName: \"kubernetes.io/projected/34d081d4-e311-40f4-9695-abcc8ff40605-kube-api-access-28mfh\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:49:52.989256 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:52.989243 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34d081d4-e311-40f4-9695-abcc8ff40605-istiod-ca-cert\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:49:52.989307 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:52.989281 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-credential-socket\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:49:52.989460 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:52.989408 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-workload-socket\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:49:52.989530 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:52.989471 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34d081d4-e311-40f4-9695-abcc8ff40605-istio-token\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:49:52.990798 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:52.990776 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-istio-data\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:49:52.991225 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:52.991201 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-workload-socket\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:49:52.991353 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:52.991333 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-credential-socket\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:49:52.991495 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:52.991474 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-workload-certs\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:49:52.991824 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:52.991805 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34d081d4-e311-40f4-9695-abcc8ff40605-istiod-ca-cert\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:49:52.993388 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:52.993364 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-istio-envoy\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:49:52.994110 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:52.994088 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34d081d4-e311-40f4-9695-abcc8ff40605-istio-podinfo\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:49:53.004314 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:53.004287 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28mfh\" (UniqueName: \"kubernetes.io/projected/34d081d4-e311-40f4-9695-abcc8ff40605-kube-api-access-28mfh\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:49:53.004743 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:53.004722 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34d081d4-e311-40f4-9695-abcc8ff40605-istio-token\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:49:53.096817 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:53.096726 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:49:53.394436 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:49:53.394405 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d081d4_e311_40f4_9695_abcc8ff40605.slice/crio-f7feafe8fee61bbb92f9f312a7ec520d6ba5969cb7fd63e7b7c532b9d6638109 WatchSource:0}: Error finding container f7feafe8fee61bbb92f9f312a7ec520d6ba5969cb7fd63e7b7c532b9d6638109: Status 404 returned error can't find the container with id f7feafe8fee61bbb92f9f312a7ec520d6ba5969cb7fd63e7b7c532b9d6638109 Nov 01 22:49:53.396077 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:53.396047 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh"] Nov 01 22:49:53.397486 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:53.397434 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:49:53.397592 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:53.397516 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:49:53.397592 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:53.397567 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:49:53.573129 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:53.573098 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2" Nov 01 22:49:53.677708 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:53.677669 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" event={"ID":"34d081d4-e311-40f4-9695-abcc8ff40605","Type":"ContainerStarted","Data":"ed3342ce267adabf14d5b803ff51e867d579d96a81c437122b619fdbc86c7428"} Nov 01 22:49:53.677947 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:53.677736 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" event={"ID":"34d081d4-e311-40f4-9695-abcc8ff40605","Type":"ContainerStarted","Data":"f7feafe8fee61bbb92f9f312a7ec520d6ba5969cb7fd63e7b7c532b9d6638109"} Nov 01 22:49:53.709333 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:53.709279 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" podStartSLOduration=1.7092623869999999 podStartE2EDuration="1.709262387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:53.705415067 +0000 UTC m=+1562.110005700" watchObservedRunningTime="2026-11-01 22:49:53.709262387 +0000 UTC m=+1562.113853018" Nov 01 22:49:54.097620 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:54.097539 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:49:54.099036 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:54.099008 5011 patch_prober.go:29] interesting pod/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 01 22:49:54.099185 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:54.099068 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" podUID="34d081d4-e311-40f4-9695-abcc8ff40605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 01 22:49:55.097514 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:55.097478 5011 patch_prober.go:29] interesting pod/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 01 22:49:55.097701 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:55.097610 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" podUID="34d081d4-e311-40f4-9695-abcc8ff40605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 01 22:49:56.098006 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:56.097965 5011 patch_prober.go:29] interesting pod/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 01 22:49:56.098425 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:56.098025 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" podUID="34d081d4-e311-40f4-9695-abcc8ff40605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 01 22:49:57.097600 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:57.097554 5011 patch_prober.go:29] interesting pod/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 01 22:49:57.097792 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:57.097620 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" podUID="34d081d4-e311-40f4-9695-abcc8ff40605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 01 22:49:58.097441 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:58.097401 5011 patch_prober.go:29] interesting pod/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 01 22:49:58.097865 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:58.097462 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" podUID="34d081d4-e311-40f4-9695-abcc8ff40605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 01 22:49:59.097520 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:59.097483 5011 patch_prober.go:29] interesting pod/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 01 22:49:59.098034 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:49:59.097561 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" podUID="34d081d4-e311-40f4-9695-abcc8ff40605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 01 22:50:00.098786 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:00.098749 5011 patch_prober.go:29] interesting pod/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 01 22:50:00.099235 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:00.098822 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" podUID="34d081d4-e311-40f4-9695-abcc8ff40605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 01 22:50:01.207783 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:01.207751 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:50:01.208353 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:01.208319 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:50:01.209667 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:01.209643 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:50:02.656263 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:02.656234 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0e685d-784cb6758c-4nwqk" Nov 01 22:50:02.656667 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:02.656314 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e685d-6f4746bcb7-l65sf" Nov 01 22:50:16.961159 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:50:16.961116 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:50:17.444953 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:17.444922 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6"] Nov 01 22:50:17.445154 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:17.445125 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" podUID="b3927dac-a03b-43fd-bdd4-6959001771f1" containerName="istio-proxy" containerID="cri-o://61efa79764cbd2db12dd90c1d828467c85fdbdaefa1dba34e75cc0dc4e53e4db" gracePeriod=30 Nov 01 22:50:22.861194 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:22.861168 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:50:22.904933 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:22.904892 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0e685d-649786988c-nx4df"] Nov 01 22:50:22.906016 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:22.905994 5011 cpu_manager.go:414] "containerMap: removing container" podUID="b3927dac-a03b-43fd-bdd4-6959001771f1" containerName="istio-proxy" Nov 01 22:50:22.906016 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:22.906012 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3927dac-a03b-43fd-bdd4-6959001771f1" containerName="istio-proxy" Nov 01 22:50:22.906217 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:22.906176 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3927dac-a03b-43fd-bdd4-6959001771f1" containerName="istio-proxy" Nov 01 22:50:22.907312 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:22.907283 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3927dac-a03b-43fd-bdd4-6959001771f1-kube-api-access-bqf8j\" (UniqueName: \"kubernetes.io/projected/b3927dac-a03b-43fd-bdd4-6959001771f1-kube-api-access-bqf8j\") pod \"b3927dac-a03b-43fd-bdd4-6959001771f1\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " Nov 01 22:50:22.907441 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:22.907337 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-credential-socket\") pod \"b3927dac-a03b-43fd-bdd4-6959001771f1\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " Nov 01 22:50:22.907555 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:22.907533 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3927dac-a03b-43fd-bdd4-6959001771f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3927dac-a03b-43fd-bdd4-6959001771f1-istiod-ca-cert\") pod \"b3927dac-a03b-43fd-bdd4-6959001771f1\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " Nov 01 22:50:22.907624 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:22.907596 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-workload-socket\") pod \"b3927dac-a03b-43fd-bdd4-6959001771f1\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " Nov 01 22:50:22.907688 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:22.907634 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-credential-socket" pod "b3927dac-a03b-43fd-bdd4-6959001771f1" (UID: "b3927dac-a03b-43fd-bdd4-6959001771f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:22.907688 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:22.907660 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3927dac-a03b-43fd-bdd4-6959001771f1-istio-token\" (UniqueName: \"kubernetes.io/projected/b3927dac-a03b-43fd-bdd4-6959001771f1-istio-token\") pod \"b3927dac-a03b-43fd-bdd4-6959001771f1\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " Nov 01 22:50:22.907889 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:22.907855 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-workload-socket" pod "b3927dac-a03b-43fd-bdd4-6959001771f1" (UID: "b3927dac-a03b-43fd-bdd4-6959001771f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:22.908021 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:22.907924 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-istio-envoy\") pod \"b3927dac-a03b-43fd-bdd4-6959001771f1\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " Nov 01 22:50:22.908021 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:22.907943 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3927dac-a03b-43fd-bdd4-6959001771f1-istiod-ca-cert" pod "b3927dac-a03b-43fd-bdd4-6959001771f1" (UID: "b3927dac-a03b-43fd-bdd4-6959001771f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:50:22.908021 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:22.907973 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-workload-certs\") pod \"b3927dac-a03b-43fd-bdd4-6959001771f1\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " Nov 01 22:50:22.908194 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:22.908131 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-istio-data\") pod \"b3927dac-a03b-43fd-bdd4-6959001771f1\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " Nov 01 22:50:22.908194 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:22.908184 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3927dac-a03b-43fd-bdd4-6959001771f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3927dac-a03b-43fd-bdd4-6959001771f1-istio-podinfo\") pod \"b3927dac-a03b-43fd-bdd4-6959001771f1\" (UID: \"b3927dac-a03b-43fd-bdd4-6959001771f1\") " Nov 01 22:50:22.908514 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:22.908461 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-workload-certs" pod "b3927dac-a03b-43fd-bdd4-6959001771f1" (UID: "b3927dac-a03b-43fd-bdd4-6959001771f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:22.908674 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:22.908646 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-istio-data" pod "b3927dac-a03b-43fd-bdd4-6959001771f1" (UID: "b3927dac-a03b-43fd-bdd4-6959001771f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:22.908978 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:22.908960 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:22.909084 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:22.908982 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:22.909084 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:22.908997 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:22.909084 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:22.909010 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3927dac-a03b-43fd-bdd4-6959001771f1-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:22.909084 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:22.909023 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:22.911199 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:22.911184 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e685d-649786988c-nx4df" Nov 01 22:50:22.912282 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:22.912248 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3927dac-a03b-43fd-bdd4-6959001771f1-istio-podinfo" pod "b3927dac-a03b-43fd-bdd4-6959001771f1" (UID: "b3927dac-a03b-43fd-bdd4-6959001771f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:50:22.918096 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:22.918067 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0e685d-649786988c-nx4df"] Nov 01 22:50:22.918475 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:22.918384 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3927dac-a03b-43fd-bdd4-6959001771f1-kube-api-access-bqf8j" pod "b3927dac-a03b-43fd-bdd4-6959001771f1" (UID: "b3927dac-a03b-43fd-bdd4-6959001771f1"). InnerVolumeSpecName "kube-api-access-bqf8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:22.922061 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:22.922032 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-istio-envoy" pod "b3927dac-a03b-43fd-bdd4-6959001771f1" (UID: "b3927dac-a03b-43fd-bdd4-6959001771f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:22.922299 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:22.922282 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3927dac-a03b-43fd-bdd4-6959001771f1-istio-token" pod "b3927dac-a03b-43fd-bdd4-6959001771f1" (UID: "b3927dac-a03b-43fd-bdd4-6959001771f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:22.922521 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:22.922504 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-0e685d-serving-cert\"" Nov 01 22:50:23.010372 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:23.010336 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vltqc\" (UniqueName: \"kubernetes.io/projected/84b76836-25e9-4bb4-9e99-fd9aa116c30e-kube-api-access-vltqc\") pod \"maas-api-e2e-inf-b-0e685d-649786988c-nx4df\" (UID: \"84b76836-25e9-4bb4-9e99-fd9aa116c30e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e685d-649786988c-nx4df" Nov 01 22:50:23.010991 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:23.010968 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84b76836-25e9-4bb4-9e99-fd9aa116c30e-metrics-certs\") pod \"maas-api-e2e-inf-b-0e685d-649786988c-nx4df\" (UID: \"84b76836-25e9-4bb4-9e99-fd9aa116c30e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e685d-649786988c-nx4df" Nov 01 22:50:23.011240 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:23.011221 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84b76836-25e9-4bb4-9e99-fd9aa116c30e-maas-api-tls\") pod \"maas-api-e2e-inf-b-0e685d-649786988c-nx4df\" (UID: \"84b76836-25e9-4bb4-9e99-fd9aa116c30e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e685d-649786988c-nx4df" Nov 01 22:50:23.011613 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:23.011596 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b76836-25e9-4bb4-9e99-fd9aa116c30e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0e685d-649786988c-nx4df\" (UID: \"84b76836-25e9-4bb4-9e99-fd9aa116c30e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e685d-649786988c-nx4df" Nov 01 22:50:23.011949 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:23.011933 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqf8j\" (UniqueName: \"kubernetes.io/projected/b3927dac-a03b-43fd-bdd4-6959001771f1-kube-api-access-bqf8j\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.012098 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:23.012087 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3927dac-a03b-43fd-bdd4-6959001771f1-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.012221 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:23.012195 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3927dac-a03b-43fd-bdd4-6959001771f1-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.012323 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:23.012311 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3927dac-a03b-43fd-bdd4-6959001771f1-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.113659 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:23.113561 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b76836-25e9-4bb4-9e99-fd9aa116c30e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0e685d-649786988c-nx4df\" (UID: \"84b76836-25e9-4bb4-9e99-fd9aa116c30e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e685d-649786988c-nx4df" Nov 01 22:50:23.113821 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:23.113691 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vltqc\" (UniqueName: \"kubernetes.io/projected/84b76836-25e9-4bb4-9e99-fd9aa116c30e-kube-api-access-vltqc\") pod \"maas-api-e2e-inf-b-0e685d-649786988c-nx4df\" (UID: \"84b76836-25e9-4bb4-9e99-fd9aa116c30e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e685d-649786988c-nx4df" Nov 01 22:50:23.113963 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:23.113888 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84b76836-25e9-4bb4-9e99-fd9aa116c30e-metrics-certs\") pod \"maas-api-e2e-inf-b-0e685d-649786988c-nx4df\" (UID: \"84b76836-25e9-4bb4-9e99-fd9aa116c30e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e685d-649786988c-nx4df" Nov 01 22:50:23.113963 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:23.113951 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84b76836-25e9-4bb4-9e99-fd9aa116c30e-maas-api-tls\") pod \"maas-api-e2e-inf-b-0e685d-649786988c-nx4df\" (UID: \"84b76836-25e9-4bb4-9e99-fd9aa116c30e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e685d-649786988c-nx4df" Nov 01 22:50:23.114369 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:23.114344 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b76836-25e9-4bb4-9e99-fd9aa116c30e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0e685d-649786988c-nx4df\" (UID: \"84b76836-25e9-4bb4-9e99-fd9aa116c30e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e685d-649786988c-nx4df" Nov 01 22:50:23.116595 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:23.116573 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84b76836-25e9-4bb4-9e99-fd9aa116c30e-maas-api-tls\") pod \"maas-api-e2e-inf-b-0e685d-649786988c-nx4df\" (UID: \"84b76836-25e9-4bb4-9e99-fd9aa116c30e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e685d-649786988c-nx4df" Nov 01 22:50:23.116674 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:23.116599 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84b76836-25e9-4bb4-9e99-fd9aa116c30e-metrics-certs\") pod \"maas-api-e2e-inf-b-0e685d-649786988c-nx4df\" (UID: \"84b76836-25e9-4bb4-9e99-fd9aa116c30e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e685d-649786988c-nx4df" Nov 01 22:50:23.122598 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:23.122566 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vltqc\" (UniqueName: \"kubernetes.io/projected/84b76836-25e9-4bb4-9e99-fd9aa116c30e-kube-api-access-vltqc\") pod \"maas-api-e2e-inf-b-0e685d-649786988c-nx4df\" (UID: \"84b76836-25e9-4bb4-9e99-fd9aa116c30e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e685d-649786988c-nx4df" Nov 01 22:50:23.265440 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:23.265412 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e685d-649786988c-nx4df" Nov 01 22:50:23.352935 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:23.352480 5011 generic.go:356] "Generic (PLEG): container finished" podID="b3927dac-a03b-43fd-bdd4-6959001771f1" containerID="61efa79764cbd2db12dd90c1d828467c85fdbdaefa1dba34e75cc0dc4e53e4db" exitCode=0 Nov 01 22:50:23.352935 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:23.352591 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" event={"ID":"b3927dac-a03b-43fd-bdd4-6959001771f1","Type":"ContainerDied","Data":"61efa79764cbd2db12dd90c1d828467c85fdbdaefa1dba34e75cc0dc4e53e4db"} Nov 01 22:50:23.352935 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:23.352645 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" event={"ID":"b3927dac-a03b-43fd-bdd4-6959001771f1","Type":"ContainerDied","Data":"e345e16628feb28eaa9396d62f52eea3a274da0742416df1acb8903fc34b3334"} Nov 01 22:50:23.352935 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:23.352674 5011 scope.go:122] "RemoveContainer" containerID="61efa79764cbd2db12dd90c1d828467c85fdbdaefa1dba34e75cc0dc4e53e4db" Nov 01 22:50:23.352935 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:23.352867 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6" Nov 01 22:50:23.375040 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:23.374307 5011 scope.go:122] "RemoveContainer" containerID="61efa79764cbd2db12dd90c1d828467c85fdbdaefa1dba34e75cc0dc4e53e4db" Nov 01 22:50:23.375040 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:50:23.374540 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61efa79764cbd2db12dd90c1d828467c85fdbdaefa1dba34e75cc0dc4e53e4db\": container with ID starting with 61efa79764cbd2db12dd90c1d828467c85fdbdaefa1dba34e75cc0dc4e53e4db not found: ID does not exist" containerID="61efa79764cbd2db12dd90c1d828467c85fdbdaefa1dba34e75cc0dc4e53e4db" Nov 01 22:50:23.375040 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:23.374582 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61efa79764cbd2db12dd90c1d828467c85fdbdaefa1dba34e75cc0dc4e53e4db"} err="failed to get container status \"61efa79764cbd2db12dd90c1d828467c85fdbdaefa1dba34e75cc0dc4e53e4db\": rpc error: code = NotFound desc = could not find container \"61efa79764cbd2db12dd90c1d828467c85fdbdaefa1dba34e75cc0dc4e53e4db\": container with ID starting with 61efa79764cbd2db12dd90c1d828467c85fdbdaefa1dba34e75cc0dc4e53e4db not found: ID does not exist" Nov 01 22:50:23.404153 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:23.404088 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6"] Nov 01 22:50:23.410687 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:23.410055 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-wjgv6"] Nov 01 22:50:23.591954 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:23.580046 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0e685d-649786988c-nx4df"] Nov 01 22:50:23.749521 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:23.749490 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3927dac-a03b-43fd-bdd4-6959001771f1" path="/var/lib/kubelet/pods/b3927dac-a03b-43fd-bdd4-6959001771f1/volumes" Nov 01 22:50:24.368706 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:24.368609 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e685d-649786988c-nx4df" event={"ID":"84b76836-25e9-4bb4-9e99-fd9aa116c30e","Type":"ContainerStarted","Data":"dd70edb64d035321229d06b783ae599e8aa24bff635931f3f4b77a68098cab07"} Nov 01 22:50:24.368706 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:24.368676 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e685d-649786988c-nx4df" event={"ID":"84b76836-25e9-4bb4-9e99-fd9aa116c30e","Type":"ContainerStarted","Data":"f2a94c3da07e54d10d8e041df91788e08e7bfd629a54f2c350386808c3121d0e"} Nov 01 22:50:24.369214 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:24.368729 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e685d-649786988c-nx4df" Nov 01 22:50:24.395080 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:24.395031 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e685d-649786988c-nx4df" podStartSLOduration=12.915621567 podStartE2EDuration="13.395016821s" totalImagesPullingTime="479.395254ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:50:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:50:24.391484181 +0000 UTC m=+1592.796074813" watchObservedRunningTime="2026-11-01 22:50:24.395016821 +0000 UTC m=+1592.799607452" Nov 01 22:50:30.440883 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:30.418076 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e685d-649786988c-nx4df" Nov 01 22:50:46.964988 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:50:46.964950 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:50:48.514032 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:48.513981 5011 patch_prober.go:29] interesting pod/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:50:48.514572 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:48.514060 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" podUID="deb96044-ee0a-4050-8076-ebce2c1d990a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:50:58.300129 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:58.300071 5011 generic.go:356] "Generic (PLEG): container finished" podID="deb96044-ee0a-4050-8076-ebce2c1d990a" containerID="f4077be34fc1ad4c8666239c81e6092a00dba3cac0dd365251f732426419cc97" exitCode=0 Nov 01 22:50:58.301029 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:58.300999 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" event={"ID":"deb96044-ee0a-4050-8076-ebce2c1d990a","Type":"ContainerDied","Data":"f4077be34fc1ad4c8666239c81e6092a00dba3cac0dd365251f732426419cc97"} Nov 01 22:50:58.302022 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:58.301974 5011 scope.go:122] "RemoveContainer" containerID="f4077be34fc1ad4c8666239c81e6092a00dba3cac0dd365251f732426419cc97" Nov 01 22:50:58.307879 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:58.307725 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:50:58.332956 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:58.332724 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:50:58.333149 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:58.333017 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:50:59.311779 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:59.311740 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" event={"ID":"deb96044-ee0a-4050-8076-ebce2c1d990a","Type":"ContainerStarted","Data":"8f5ae1bd5a2431219ec22ebc3f97d5b7c8d871b2f6278cf73bbba06c71ca1679"} Nov 01 22:50:59.450539 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:59.450505 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:50:59.466432 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:59.466395 5011 patch_prober.go:29] interesting pod/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 01 22:50:59.466760 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:50:59.466722 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" podUID="deb96044-ee0a-4050-8076-ebce2c1d990a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 01 22:51:00.461857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:51:00.461247 5011 patch_prober.go:29] interesting pod/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 01 22:51:00.461857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:51:00.461307 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" podUID="deb96044-ee0a-4050-8076-ebce2c1d990a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 01 22:51:01.456919 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:51:01.446225 5011 patch_prober.go:29] interesting pod/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 01 22:51:01.456919 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:51:01.446298 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" podUID="deb96044-ee0a-4050-8076-ebce2c1d990a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 01 22:51:02.466996 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:51:02.466955 5011 patch_prober.go:29] interesting pod/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 01 22:51:02.467619 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:51:02.467585 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" podUID="deb96044-ee0a-4050-8076-ebce2c1d990a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 01 22:51:02.472627 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:51:02.472599 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:51:03.448856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:51:03.446248 5011 patch_prober.go:29] interesting pod/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 01 22:51:03.448856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:51:03.446312 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" podUID="deb96044-ee0a-4050-8076-ebce2c1d990a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 01 22:51:04.448862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:51:04.444788 5011 patch_prober.go:29] interesting pod/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Nov 01 22:51:04.448862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:51:04.444868 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" podUID="deb96044-ee0a-4050-8076-ebce2c1d990a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Nov 01 22:51:05.449211 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:51:05.449180 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:51:05.450439 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:51:05.450418 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 22:51:16.963746 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:51:16.963701 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:51:32.817085 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:51:32.817047 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:51:36.015183 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:51:36.015132 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:36.016355 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:51:36.016232 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:47.069686 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:51:47.069650 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:51:55.289278 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:51:55.289239 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx"] Nov 01 22:51:55.289804 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:51:55.289491 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" podUID="02d4e19a-0aea-4d86-9876-23181ac12a4a" containerName="istio-proxy" containerID="cri-o://6708ab383780e34dbac979eadf2f9ac8fc511ecf01de2e99700f37c7f0d364a2" gracePeriod=30 Nov 01 22:51:55.294218 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:51:55.294189 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb"] Nov 01 22:51:55.294400 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:51:55.294378 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" podUID="10f200df-709a-461d-b36c-af5003f8857a" containerName="istio-proxy" containerID="cri-o://e26f2bfa4253e58d6629aa228ffc20d6b5b450973210b659f223520c8c164596" gracePeriod=30 Nov 01 22:51:55.608210 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:51:55.608111 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn"] Nov 01 22:51:55.608419 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:51:55.608388 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" podUID="068185b8-d68f-4de6-98da-8997ac7dc6db" containerName="istio-proxy" containerID="cri-o://96ca5830c624be2beed3560a891485f709c398aa7011c93c29be861aea7a7215" gracePeriod=30 Nov 01 22:51:56.604891 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:51:56.604856 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:51:56.605400 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:51:56.605363 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" podUID="02d4e19a-0aea-4d86-9876-23181ac12a4a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:51:57.577926 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:51:57.577879 5011 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:51:57.578137 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:51:57.577940 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" podUID="10f200df-709a-461d-b36c-af5003f8857a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:51:58.416631 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:51:58.416598 5011 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:51:58.417057 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:51:58.416661 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" podUID="068185b8-d68f-4de6-98da-8997ac7dc6db" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:52:00.632693 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.632649 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:52:00.646315 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.646274 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02d4e19a-0aea-4d86-9876-23181ac12a4a-istio-token\" (UniqueName: \"kubernetes.io/projected/02d4e19a-0aea-4d86-9876-23181ac12a4a-istio-token\") pod \"02d4e19a-0aea-4d86-9876-23181ac12a4a\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " Nov 01 22:52:00.646476 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.646416 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02d4e19a-0aea-4d86-9876-23181ac12a4a-kube-api-access-lzt88\" (UniqueName: \"kubernetes.io/projected/02d4e19a-0aea-4d86-9876-23181ac12a4a-kube-api-access-lzt88\") pod \"02d4e19a-0aea-4d86-9876-23181ac12a4a\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " Nov 01 22:52:00.646539 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.646494 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-istio-envoy\") pod \"02d4e19a-0aea-4d86-9876-23181ac12a4a\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " Nov 01 22:52:00.646601 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.646578 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-credential-socket\") pod \"02d4e19a-0aea-4d86-9876-23181ac12a4a\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " Nov 01 22:52:00.646682 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.646667 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-workload-certs\") pod \"02d4e19a-0aea-4d86-9876-23181ac12a4a\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " Nov 01 22:52:00.646759 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.646743 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02d4e19a-0aea-4d86-9876-23181ac12a4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d4e19a-0aea-4d86-9876-23181ac12a4a-istiod-ca-cert\") pod \"02d4e19a-0aea-4d86-9876-23181ac12a4a\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " Nov 01 22:52:00.646900 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.646882 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-workload-socket\") pod \"02d4e19a-0aea-4d86-9876-23181ac12a4a\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " Nov 01 22:52:00.647010 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.646934 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02d4e19a-0aea-4d86-9876-23181ac12a4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d4e19a-0aea-4d86-9876-23181ac12a4a-istio-podinfo\") pod \"02d4e19a-0aea-4d86-9876-23181ac12a4a\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " Nov 01 22:52:00.647010 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.646966 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-istio-data\") pod \"02d4e19a-0aea-4d86-9876-23181ac12a4a\" (UID: \"02d4e19a-0aea-4d86-9876-23181ac12a4a\") " Nov 01 22:52:00.647132 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.647034 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-credential-socket" pod "02d4e19a-0aea-4d86-9876-23181ac12a4a" (UID: "02d4e19a-0aea-4d86-9876-23181ac12a4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:00.647331 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.647312 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-workload-socket" pod "02d4e19a-0aea-4d86-9876-23181ac12a4a" (UID: "02d4e19a-0aea-4d86-9876-23181ac12a4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:00.647579 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.647538 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-istio-data" pod "02d4e19a-0aea-4d86-9876-23181ac12a4a" (UID: "02d4e19a-0aea-4d86-9876-23181ac12a4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:00.647741 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.647724 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-workload-certs" pod "02d4e19a-0aea-4d86-9876-23181ac12a4a" (UID: "02d4e19a-0aea-4d86-9876-23181ac12a4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:00.647806 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.647649 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02d4e19a-0aea-4d86-9876-23181ac12a4a-istiod-ca-cert" pod "02d4e19a-0aea-4d86-9876-23181ac12a4a" (UID: "02d4e19a-0aea-4d86-9876-23181ac12a4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:52:00.647933 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.647915 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:00.647990 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.647940 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:00.647990 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.647954 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:00.659823 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.659481 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d4e19a-0aea-4d86-9876-23181ac12a4a-istio-token" pod "02d4e19a-0aea-4d86-9876-23181ac12a4a" (UID: "02d4e19a-0aea-4d86-9876-23181ac12a4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:52:00.659823 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.659598 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-istio-envoy" pod "02d4e19a-0aea-4d86-9876-23181ac12a4a" (UID: "02d4e19a-0aea-4d86-9876-23181ac12a4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:00.661421 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.661391 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d4e19a-0aea-4d86-9876-23181ac12a4a-kube-api-access-lzt88" pod "02d4e19a-0aea-4d86-9876-23181ac12a4a" (UID: "02d4e19a-0aea-4d86-9876-23181ac12a4a"). InnerVolumeSpecName "kube-api-access-lzt88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:52:00.661614 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.661587 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02d4e19a-0aea-4d86-9876-23181ac12a4a-istio-podinfo" pod "02d4e19a-0aea-4d86-9876-23181ac12a4a" (UID: "02d4e19a-0aea-4d86-9876-23181ac12a4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:52:00.673191 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.673159 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp"] Nov 01 22:52:00.674487 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.674461 5011 cpu_manager.go:414] "containerMap: removing container" podUID="02d4e19a-0aea-4d86-9876-23181ac12a4a" containerName="istio-proxy" Nov 01 22:52:00.674576 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.674491 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02d4e19a-0aea-4d86-9876-23181ac12a4a" containerName="istio-proxy" Nov 01 22:52:00.674576 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.674559 5011 cpu_manager.go:414] "containerMap: removing container" podUID="02d4e19a-0aea-4d86-9876-23181ac12a4a" containerName="istio-proxy" Nov 01 22:52:00.674576 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.674567 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02d4e19a-0aea-4d86-9876-23181ac12a4a" containerName="istio-proxy" Nov 01 22:52:00.674817 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.674802 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="02d4e19a-0aea-4d86-9876-23181ac12a4a" containerName="istio-proxy" Nov 01 22:52:00.681029 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.681003 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:00.685288 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.685256 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6b78bd-openshift-default-dockercfg-hv5s7\"" Nov 01 22:52:00.691228 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.691203 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp"] Nov 01 22:52:00.732008 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.731979 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:52:00.749248 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.749215 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10f200df-709a-461d-b36c-af5003f8857a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f200df-709a-461d-b36c-af5003f8857a-istiod-ca-cert\") pod \"10f200df-709a-461d-b36c-af5003f8857a\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " Nov 01 22:52:00.749437 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.749271 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-istio-data\") pod \"10f200df-709a-461d-b36c-af5003f8857a\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " Nov 01 22:52:00.749437 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.749329 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-workload-socket\") pod \"10f200df-709a-461d-b36c-af5003f8857a\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " Nov 01 22:52:00.749437 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.749370 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10f200df-709a-461d-b36c-af5003f8857a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f200df-709a-461d-b36c-af5003f8857a-istio-podinfo\") pod \"10f200df-709a-461d-b36c-af5003f8857a\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " Nov 01 22:52:00.749437 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.749419 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f200df-709a-461d-b36c-af5003f8857a-istio-token\" (UniqueName: \"kubernetes.io/projected/10f200df-709a-461d-b36c-af5003f8857a-istio-token\") pod \"10f200df-709a-461d-b36c-af5003f8857a\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " Nov 01 22:52:00.749656 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.749522 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-istio-envoy\") pod \"10f200df-709a-461d-b36c-af5003f8857a\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " Nov 01 22:52:00.749656 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.749607 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-workload-certs\") pod \"10f200df-709a-461d-b36c-af5003f8857a\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " Nov 01 22:52:00.749759 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.749674 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f200df-709a-461d-b36c-af5003f8857a-kube-api-access-8c49c\" (UniqueName: \"kubernetes.io/projected/10f200df-709a-461d-b36c-af5003f8857a-kube-api-access-8c49c\") pod \"10f200df-709a-461d-b36c-af5003f8857a\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " Nov 01 22:52:00.749812 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.749759 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10f200df-709a-461d-b36c-af5003f8857a-istiod-ca-cert" pod "10f200df-709a-461d-b36c-af5003f8857a" (UID: "10f200df-709a-461d-b36c-af5003f8857a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:52:00.749903 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.749809 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-workload-socket" pod "10f200df-709a-461d-b36c-af5003f8857a" (UID: "10f200df-709a-461d-b36c-af5003f8857a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:00.749962 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.749939 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-credential-socket\") pod \"10f200df-709a-461d-b36c-af5003f8857a\" (UID: \"10f200df-709a-461d-b36c-af5003f8857a\") " Nov 01 22:52:00.750227 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.750177 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-istio-data" pod "10f200df-709a-461d-b36c-af5003f8857a" (UID: "10f200df-709a-461d-b36c-af5003f8857a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:00.750336 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.750259 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/831636af-90aa-41a1-b03f-561c0462ab4e-istio-podinfo\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:00.750336 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.750306 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-workload-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:00.750695 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.750485 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/831636af-90aa-41a1-b03f-561c0462ab4e-istiod-ca-cert\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:00.750695 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.750600 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-workload-certs" pod "10f200df-709a-461d-b36c-af5003f8857a" (UID: "10f200df-709a-461d-b36c-af5003f8857a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:00.750987 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.750766 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-workload-certs\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:00.750987 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.750794 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/831636af-90aa-41a1-b03f-561c0462ab4e-istio-token\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:00.750987 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.750856 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-credential-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:00.751232 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.751213 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-istio-data\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:00.751474 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.751454 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-credential-socket" pod "10f200df-709a-461d-b36c-af5003f8857a" (UID: "10f200df-709a-461d-b36c-af5003f8857a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:00.751732 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.751711 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7cwd\" (UniqueName: \"kubernetes.io/projected/831636af-90aa-41a1-b03f-561c0462ab4e-kube-api-access-v7cwd\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:00.752003 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.751771 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-istio-envoy\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:00.752100 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.752052 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f200df-709a-461d-b36c-af5003f8857a-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:00.752336 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.752317 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:00.752428 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.752346 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzt88\" (UniqueName: \"kubernetes.io/projected/02d4e19a-0aea-4d86-9876-23181ac12a4a-kube-api-access-lzt88\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:00.752428 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.752365 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:00.752428 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.752380 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:00.752428 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.752395 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d4e19a-0aea-4d86-9876-23181ac12a4a-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:00.752618 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.752459 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d4e19a-0aea-4d86-9876-23181ac12a4a-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:00.752618 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.752481 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:00.752618 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.752499 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d4e19a-0aea-4d86-9876-23181ac12a4a-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:00.752618 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.752514 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:00.752618 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.752528 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d4e19a-0aea-4d86-9876-23181ac12a4a-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:00.753008 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.752988 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-istio-envoy" pod "10f200df-709a-461d-b36c-af5003f8857a" (UID: "10f200df-709a-461d-b36c-af5003f8857a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:00.755343 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.755311 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f200df-709a-461d-b36c-af5003f8857a-kube-api-access-8c49c" pod "10f200df-709a-461d-b36c-af5003f8857a" (UID: "10f200df-709a-461d-b36c-af5003f8857a"). InnerVolumeSpecName "kube-api-access-8c49c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:52:00.756170 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.756141 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f200df-709a-461d-b36c-af5003f8857a-istio-token" pod "10f200df-709a-461d-b36c-af5003f8857a" (UID: "10f200df-709a-461d-b36c-af5003f8857a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:52:00.757198 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.757158 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10f200df-709a-461d-b36c-af5003f8857a-istio-podinfo" pod "10f200df-709a-461d-b36c-af5003f8857a" (UID: "10f200df-709a-461d-b36c-af5003f8857a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:52:00.783326 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.783272 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr"] Nov 01 22:52:00.785153 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.785122 5011 cpu_manager.go:414] "containerMap: removing container" podUID="10f200df-709a-461d-b36c-af5003f8857a" containerName="istio-proxy" Nov 01 22:52:00.785363 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.785345 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f200df-709a-461d-b36c-af5003f8857a" containerName="istio-proxy" Nov 01 22:52:00.786028 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.786006 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="02d4e19a-0aea-4d86-9876-23181ac12a4a" containerName="istio-proxy" Nov 01 22:52:00.786171 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.786159 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f200df-709a-461d-b36c-af5003f8857a" containerName="istio-proxy" Nov 01 22:52:00.793565 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.793546 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:00.800198 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.800139 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr"] Nov 01 22:52:00.854889 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.854795 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/831636af-90aa-41a1-b03f-561c0462ab4e-istio-podinfo\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:00.855953 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.855174 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-workload-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:00.855953 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.855226 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-workload-certs\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:00.855953 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.855346 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istio-data\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:00.855953 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.855411 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istio-podinfo\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:00.855953 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.855436 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istio-token\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:00.855953 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.855470 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/831636af-90aa-41a1-b03f-561c0462ab4e-istiod-ca-cert\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:00.855953 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.855509 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istio-envoy\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:00.855953 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.855534 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtggj\" (UniqueName: \"kubernetes.io/projected/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-kube-api-access-xtggj\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:00.855953 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.855663 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-workload-certs\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:00.855953 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.855688 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/831636af-90aa-41a1-b03f-561c0462ab4e-istio-token\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:00.855953 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.855716 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-workload-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:00.855953 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.855758 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-credential-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:00.855953 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.855801 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-istio-data\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:00.858054 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.856845 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-credential-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:00.858054 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.856907 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7cwd\" (UniqueName: \"kubernetes.io/projected/831636af-90aa-41a1-b03f-561c0462ab4e-kube-api-access-v7cwd\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:00.858054 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.856945 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istiod-ca-cert\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:00.858054 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.856999 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-istio-envoy\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:00.858054 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.857202 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f200df-709a-461d-b36c-af5003f8857a-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:00.858054 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.857219 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f200df-709a-461d-b36c-af5003f8857a-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:00.858054 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.857233 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f200df-709a-461d-b36c-af5003f8857a-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:00.858054 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.857248 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c49c\" (UniqueName: \"kubernetes.io/projected/10f200df-709a-461d-b36c-af5003f8857a-kube-api-access-8c49c\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:00.859293 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.859266 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-workload-certs\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:00.860039 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.860014 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-workload-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:00.861123 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.860720 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/831636af-90aa-41a1-b03f-561c0462ab4e-istiod-ca-cert\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:00.861231 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.861134 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/831636af-90aa-41a1-b03f-561c0462ab4e-istio-podinfo\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:00.861647 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.861624 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-istio-data\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:00.861778 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.861750 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-credential-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:00.864398 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.864372 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-istio-envoy\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:00.902899 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.897378 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/831636af-90aa-41a1-b03f-561c0462ab4e-istio-token\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:00.951856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.931545 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7cwd\" (UniqueName: \"kubernetes.io/projected/831636af-90aa-41a1-b03f-561c0462ab4e-kube-api-access-v7cwd\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:00.985858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.980119 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istio-data\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:00.985858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.980219 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istio-podinfo\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:00.985858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.980249 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istio-token\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:00.985858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.980296 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istio-envoy\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:00.985858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.980326 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtggj\" (UniqueName: \"kubernetes.io/projected/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-kube-api-access-xtggj\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:00.985858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.980466 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-workload-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:00.985858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.980612 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-credential-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:00.985858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.980665 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istiod-ca-cert\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:00.985858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.980812 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-workload-certs\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:00.985858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.981731 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-workload-certs\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:00.985858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.982144 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istio-data\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:00.985858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.983723 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-credential-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:00.985858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.983987 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-workload-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:00.985858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:00.984692 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istiod-ca-cert\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:01.012662 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.012585 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istio-podinfo\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:01.014373 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.013337 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istio-token\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:01.014973 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.014920 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istio-envoy\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:01.018341 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.018316 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtggj\" (UniqueName: \"kubernetes.io/projected/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-kube-api-access-xtggj\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:01.030128 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.030105 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:01.050115 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.050071 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:52:01.087293 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.087253 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-credential-socket\") pod \"068185b8-d68f-4de6-98da-8997ac7dc6db\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " Nov 01 22:52:01.087487 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.087327 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-workload-certs\") pod \"068185b8-d68f-4de6-98da-8997ac7dc6db\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " Nov 01 22:52:01.087487 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.087374 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/068185b8-d68f-4de6-98da-8997ac7dc6db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/068185b8-d68f-4de6-98da-8997ac7dc6db-istio-podinfo\") pod \"068185b8-d68f-4de6-98da-8997ac7dc6db\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " Nov 01 22:52:01.087487 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.087457 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/068185b8-d68f-4de6-98da-8997ac7dc6db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/068185b8-d68f-4de6-98da-8997ac7dc6db-istiod-ca-cert\") pod \"068185b8-d68f-4de6-98da-8997ac7dc6db\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " Nov 01 22:52:01.087654 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.087518 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/068185b8-d68f-4de6-98da-8997ac7dc6db-kube-api-access-6crmw\" (UniqueName: \"kubernetes.io/projected/068185b8-d68f-4de6-98da-8997ac7dc6db-kube-api-access-6crmw\") pod \"068185b8-d68f-4de6-98da-8997ac7dc6db\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " Nov 01 22:52:01.087654 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.087593 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-istio-data\") pod \"068185b8-d68f-4de6-98da-8997ac7dc6db\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " Nov 01 22:52:01.087654 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.087632 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/068185b8-d68f-4de6-98da-8997ac7dc6db-istio-token\" (UniqueName: \"kubernetes.io/projected/068185b8-d68f-4de6-98da-8997ac7dc6db-istio-token\") pod \"068185b8-d68f-4de6-98da-8997ac7dc6db\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " Nov 01 22:52:01.087808 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.087769 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-workload-socket\") pod \"068185b8-d68f-4de6-98da-8997ac7dc6db\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " Nov 01 22:52:01.087978 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.087960 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-istio-envoy\") pod \"068185b8-d68f-4de6-98da-8997ac7dc6db\" (UID: \"068185b8-d68f-4de6-98da-8997ac7dc6db\") " Nov 01 22:52:01.089725 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.089693 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/068185b8-d68f-4de6-98da-8997ac7dc6db-istiod-ca-cert" pod "068185b8-d68f-4de6-98da-8997ac7dc6db" (UID: "068185b8-d68f-4de6-98da-8997ac7dc6db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:52:01.090044 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.090020 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-credential-socket" pod "068185b8-d68f-4de6-98da-8997ac7dc6db" (UID: "068185b8-d68f-4de6-98da-8997ac7dc6db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:01.090260 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.090222 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-workload-certs" pod "068185b8-d68f-4de6-98da-8997ac7dc6db" (UID: "068185b8-d68f-4de6-98da-8997ac7dc6db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:01.091948 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.091514 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-istio-data" pod "068185b8-d68f-4de6-98da-8997ac7dc6db" (UID: "068185b8-d68f-4de6-98da-8997ac7dc6db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:01.092737 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.092708 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-workload-socket" pod "068185b8-d68f-4de6-98da-8997ac7dc6db" (UID: "068185b8-d68f-4de6-98da-8997ac7dc6db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:01.092737 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.092730 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns"] Nov 01 22:52:01.093237 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.093213 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:01.093323 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.093242 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:01.093323 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.093256 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/068185b8-d68f-4de6-98da-8997ac7dc6db-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:01.093323 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.093271 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:01.093323 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.093286 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:01.094003 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.093982 5011 cpu_manager.go:414] "containerMap: removing container" podUID="068185b8-d68f-4de6-98da-8997ac7dc6db" containerName="istio-proxy" Nov 01 22:52:01.094003 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.094003 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="068185b8-d68f-4de6-98da-8997ac7dc6db" containerName="istio-proxy" Nov 01 22:52:01.094162 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.094124 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="068185b8-d68f-4de6-98da-8997ac7dc6db" containerName="istio-proxy" Nov 01 22:52:01.094219 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.094194 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/068185b8-d68f-4de6-98da-8997ac7dc6db-kube-api-access-6crmw" pod "068185b8-d68f-4de6-98da-8997ac7dc6db" (UID: "068185b8-d68f-4de6-98da-8997ac7dc6db"). InnerVolumeSpecName "kube-api-access-6crmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:52:01.095617 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.095579 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-istio-envoy" pod "068185b8-d68f-4de6-98da-8997ac7dc6db" (UID: "068185b8-d68f-4de6-98da-8997ac7dc6db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:01.096465 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.096419 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/068185b8-d68f-4de6-98da-8997ac7dc6db-istio-token" pod "068185b8-d68f-4de6-98da-8997ac7dc6db" (UID: "068185b8-d68f-4de6-98da-8997ac7dc6db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:52:01.097607 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.097580 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/068185b8-d68f-4de6-98da-8997ac7dc6db-istio-podinfo" pod "068185b8-d68f-4de6-98da-8997ac7dc6db" (UID: "068185b8-d68f-4de6-98da-8997ac7dc6db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:52:01.100770 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.100751 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:01.116215 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.116185 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:01.124750 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.124719 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns"] Nov 01 22:52:01.199220 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.199181 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vl26\" (UniqueName: \"kubernetes.io/projected/00381713-efff-4f16-8fc8-af879b07e517-kube-api-access-9vl26\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:01.199426 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.199270 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-istio-data\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:01.199426 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.199422 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-workload-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:01.199526 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.199475 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-credential-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:01.199604 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.199589 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00381713-efff-4f16-8fc8-af879b07e517-istio-podinfo\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:01.199669 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.199655 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-workload-certs\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:01.199777 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.199760 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-istio-envoy\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:01.199867 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.199808 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00381713-efff-4f16-8fc8-af879b07e517-istio-token\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:01.199933 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.199864 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00381713-efff-4f16-8fc8-af879b07e517-istiod-ca-cert\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:01.202654 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.200019 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/068185b8-d68f-4de6-98da-8997ac7dc6db-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:01.202654 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.200043 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/068185b8-d68f-4de6-98da-8997ac7dc6db-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:01.202654 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.200061 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6crmw\" (UniqueName: \"kubernetes.io/projected/068185b8-d68f-4de6-98da-8997ac7dc6db-kube-api-access-6crmw\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:01.202654 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.200077 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/068185b8-d68f-4de6-98da-8997ac7dc6db-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:01.270314 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.268414 5011 generic.go:356] "Generic (PLEG): container finished" podID="10f200df-709a-461d-b36c-af5003f8857a" containerID="e26f2bfa4253e58d6629aa228ffc20d6b5b450973210b659f223520c8c164596" exitCode=0 Nov 01 22:52:01.270314 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.268491 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" Nov 01 22:52:01.270314 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.268532 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" event={"ID":"10f200df-709a-461d-b36c-af5003f8857a","Type":"ContainerDied","Data":"e26f2bfa4253e58d6629aa228ffc20d6b5b450973210b659f223520c8c164596"} Nov 01 22:52:01.270314 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.268570 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb" event={"ID":"10f200df-709a-461d-b36c-af5003f8857a","Type":"ContainerDied","Data":"7324ea0bb565f61d6aecae42333b8507d62b46a1af4f3c5fcdb953790415e2b7"} Nov 01 22:52:01.270314 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.268592 5011 scope.go:122] "RemoveContainer" containerID="e26f2bfa4253e58d6629aa228ffc20d6b5b450973210b659f223520c8c164596" Nov 01 22:52:01.272965 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.271846 5011 generic.go:356] "Generic (PLEG): container finished" podID="068185b8-d68f-4de6-98da-8997ac7dc6db" containerID="96ca5830c624be2beed3560a891485f709c398aa7011c93c29be861aea7a7215" exitCode=0 Nov 01 22:52:01.272965 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.272086 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" event={"ID":"068185b8-d68f-4de6-98da-8997ac7dc6db","Type":"ContainerDied","Data":"96ca5830c624be2beed3560a891485f709c398aa7011c93c29be861aea7a7215"} Nov 01 22:52:01.272965 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.272136 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" event={"ID":"068185b8-d68f-4de6-98da-8997ac7dc6db","Type":"ContainerDied","Data":"581789e06f96c3b7a4db6513c92ea3fed56d68de13878d68c5d0864ec1f4d2d3"} Nov 01 22:52:01.272965 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.272212 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn" Nov 01 22:52:01.278923 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.278308 5011 generic.go:356] "Generic (PLEG): container finished" podID="02d4e19a-0aea-4d86-9876-23181ac12a4a" containerID="6708ab383780e34dbac979eadf2f9ac8fc511ecf01de2e99700f37c7f0d364a2" exitCode=0 Nov 01 22:52:01.278923 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.278390 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" event={"ID":"02d4e19a-0aea-4d86-9876-23181ac12a4a","Type":"ContainerDied","Data":"6708ab383780e34dbac979eadf2f9ac8fc511ecf01de2e99700f37c7f0d364a2"} Nov 01 22:52:01.278923 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.278425 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" event={"ID":"02d4e19a-0aea-4d86-9876-23181ac12a4a","Type":"ContainerDied","Data":"f884a9904d9b1fddcf9526185f9031d9e799e43c913d40d85b379e5d1ce707ef"} Nov 01 22:52:01.278923 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.278482 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx" Nov 01 22:52:01.289543 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.289520 5011 scope.go:122] "RemoveContainer" containerID="e26f2bfa4253e58d6629aa228ffc20d6b5b450973210b659f223520c8c164596" Nov 01 22:52:01.289966 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:52:01.289944 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e26f2bfa4253e58d6629aa228ffc20d6b5b450973210b659f223520c8c164596\": container with ID starting with e26f2bfa4253e58d6629aa228ffc20d6b5b450973210b659f223520c8c164596 not found: ID does not exist" containerID="e26f2bfa4253e58d6629aa228ffc20d6b5b450973210b659f223520c8c164596" Nov 01 22:52:01.290059 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.289973 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e26f2bfa4253e58d6629aa228ffc20d6b5b450973210b659f223520c8c164596"} err="failed to get container status \"e26f2bfa4253e58d6629aa228ffc20d6b5b450973210b659f223520c8c164596\": rpc error: code = NotFound desc = could not find container \"e26f2bfa4253e58d6629aa228ffc20d6b5b450973210b659f223520c8c164596\": container with ID starting with e26f2bfa4253e58d6629aa228ffc20d6b5b450973210b659f223520c8c164596 not found: ID does not exist" Nov 01 22:52:01.290059 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.289990 5011 scope.go:122] "RemoveContainer" containerID="96ca5830c624be2beed3560a891485f709c398aa7011c93c29be861aea7a7215" Nov 01 22:52:01.302726 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.301403 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-workload-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:01.302726 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.301472 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-credential-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:01.302726 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.301585 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00381713-efff-4f16-8fc8-af879b07e517-istio-podinfo\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:01.302726 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.301638 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-workload-certs\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:01.302726 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.301734 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-istio-envoy\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:01.302726 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.301774 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00381713-efff-4f16-8fc8-af879b07e517-istio-token\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:01.302726 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.301809 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00381713-efff-4f16-8fc8-af879b07e517-istiod-ca-cert\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:01.302726 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.301916 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vl26\" (UniqueName: \"kubernetes.io/projected/00381713-efff-4f16-8fc8-af879b07e517-kube-api-access-9vl26\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:01.302726 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.301972 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-istio-data\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:01.302726 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.302326 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-workload-certs\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:01.302726 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.302398 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-istio-data\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:01.302726 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.302580 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-workload-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:01.303530 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.302815 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-credential-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:01.310230 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.303722 5011 scope.go:122] "RemoveContainer" containerID="96ca5830c624be2beed3560a891485f709c398aa7011c93c29be861aea7a7215" Nov 01 22:52:01.310230 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.303934 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb"] Nov 01 22:52:01.310230 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.304512 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00381713-efff-4f16-8fc8-af879b07e517-istiod-ca-cert\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:01.310230 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:52:01.306093 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96ca5830c624be2beed3560a891485f709c398aa7011c93c29be861aea7a7215\": container with ID starting with 96ca5830c624be2beed3560a891485f709c398aa7011c93c29be861aea7a7215 not found: ID does not exist" containerID="96ca5830c624be2beed3560a891485f709c398aa7011c93c29be861aea7a7215" Nov 01 22:52:01.310230 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.306124 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96ca5830c624be2beed3560a891485f709c398aa7011c93c29be861aea7a7215"} err="failed to get container status \"96ca5830c624be2beed3560a891485f709c398aa7011c93c29be861aea7a7215\": rpc error: code = NotFound desc = could not find container \"96ca5830c624be2beed3560a891485f709c398aa7011c93c29be861aea7a7215\": container with ID starting with 96ca5830c624be2beed3560a891485f709c398aa7011c93c29be861aea7a7215 not found: ID does not exist" Nov 01 22:52:01.310230 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.306145 5011 scope.go:122] "RemoveContainer" containerID="6708ab383780e34dbac979eadf2f9ac8fc511ecf01de2e99700f37c7f0d364a2" Nov 01 22:52:01.310230 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.308577 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-istio-envoy\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:01.311598 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.311204 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00381713-efff-4f16-8fc8-af879b07e517-istio-podinfo\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:01.313282 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.313261 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-48pvb"] Nov 01 22:52:01.318234 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.318184 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vl26\" (UniqueName: \"kubernetes.io/projected/00381713-efff-4f16-8fc8-af879b07e517-kube-api-access-9vl26\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:01.322317 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.321634 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00381713-efff-4f16-8fc8-af879b07e517-istio-token\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:01.333862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.333797 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn"] Nov 01 22:52:01.355314 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.355273 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-9jzwn"] Nov 01 22:52:01.363331 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.362876 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx"] Nov 01 22:52:01.369126 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.369090 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9gdnx"] Nov 01 22:52:01.376926 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.376871 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:52:01.377036 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.376974 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:52:01.377036 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.377006 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:52:01.377190 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.377168 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp"] Nov 01 22:52:01.425083 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.425050 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:01.445460 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.445434 5011 scope.go:122] "RemoveContainer" containerID="a9c4ce39758d1535eabde8e9d66a55bc4feeed6d5022437cdcbf75a485cb5887" Nov 01 22:52:01.457447 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.457419 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr"] Nov 01 22:52:01.457652 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:52:01.457622 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee08f7f3_27a6_44fc_9099_dbe8459e6b12.slice/crio-9f642835dfb492dae66015066b3d1e008807919cd26b73511d145da33f2c8a95 WatchSource:0}: Error finding container 9f642835dfb492dae66015066b3d1e008807919cd26b73511d145da33f2c8a95: Status 404 returned error can't find the container with id 9f642835dfb492dae66015066b3d1e008807919cd26b73511d145da33f2c8a95 Nov 01 22:52:01.461942 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.461896 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:52:01.462025 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.461990 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:52:01.462083 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.462066 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:52:01.484890 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.484868 5011 scope.go:122] "RemoveContainer" containerID="6708ab383780e34dbac979eadf2f9ac8fc511ecf01de2e99700f37c7f0d364a2" Nov 01 22:52:01.485391 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:52:01.485364 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6708ab383780e34dbac979eadf2f9ac8fc511ecf01de2e99700f37c7f0d364a2\": container with ID starting with 6708ab383780e34dbac979eadf2f9ac8fc511ecf01de2e99700f37c7f0d364a2 not found: ID does not exist" containerID="6708ab383780e34dbac979eadf2f9ac8fc511ecf01de2e99700f37c7f0d364a2" Nov 01 22:52:01.485525 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.485403 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6708ab383780e34dbac979eadf2f9ac8fc511ecf01de2e99700f37c7f0d364a2"} err="failed to get container status \"6708ab383780e34dbac979eadf2f9ac8fc511ecf01de2e99700f37c7f0d364a2\": rpc error: code = NotFound desc = could not find container \"6708ab383780e34dbac979eadf2f9ac8fc511ecf01de2e99700f37c7f0d364a2\": container with ID starting with 6708ab383780e34dbac979eadf2f9ac8fc511ecf01de2e99700f37c7f0d364a2 not found: ID does not exist" Nov 01 22:52:01.485525 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.485422 5011 scope.go:122] "RemoveContainer" containerID="a9c4ce39758d1535eabde8e9d66a55bc4feeed6d5022437cdcbf75a485cb5887" Nov 01 22:52:01.485722 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:52:01.485704 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9c4ce39758d1535eabde8e9d66a55bc4feeed6d5022437cdcbf75a485cb5887\": container with ID starting with a9c4ce39758d1535eabde8e9d66a55bc4feeed6d5022437cdcbf75a485cb5887 not found: ID does not exist" containerID="a9c4ce39758d1535eabde8e9d66a55bc4feeed6d5022437cdcbf75a485cb5887" Nov 01 22:52:01.485811 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.485729 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9c4ce39758d1535eabde8e9d66a55bc4feeed6d5022437cdcbf75a485cb5887"} err="failed to get container status \"a9c4ce39758d1535eabde8e9d66a55bc4feeed6d5022437cdcbf75a485cb5887\": rpc error: code = NotFound desc = could not find container \"a9c4ce39758d1535eabde8e9d66a55bc4feeed6d5022437cdcbf75a485cb5887\": container with ID starting with a9c4ce39758d1535eabde8e9d66a55bc4feeed6d5022437cdcbf75a485cb5887 not found: ID does not exist" Nov 01 22:52:01.771553 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.771462 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02d4e19a-0aea-4d86-9876-23181ac12a4a" path="/var/lib/kubelet/pods/02d4e19a-0aea-4d86-9876-23181ac12a4a/volumes" Nov 01 22:52:01.773060 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.773022 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="068185b8-d68f-4de6-98da-8997ac7dc6db" path="/var/lib/kubelet/pods/068185b8-d68f-4de6-98da-8997ac7dc6db/volumes" Nov 01 22:52:01.774036 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.774012 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10f200df-709a-461d-b36c-af5003f8857a" path="/var/lib/kubelet/pods/10f200df-709a-461d-b36c-af5003f8857a/volumes" Nov 01 22:52:01.787027 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:52:01.786825 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00381713_efff_4f16_8fc8_af879b07e517.slice/crio-1ba7fca4cf6bbba41b5676b7624d04b8a0a01a310b0161d002fe8229db9c34e5 WatchSource:0}: Error finding container 1ba7fca4cf6bbba41b5676b7624d04b8a0a01a310b0161d002fe8229db9c34e5: Status 404 returned error can't find the container with id 1ba7fca4cf6bbba41b5676b7624d04b8a0a01a310b0161d002fe8229db9c34e5 Nov 01 22:52:01.787217 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.787187 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns"] Nov 01 22:52:01.791258 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.791218 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:52:01.791367 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.791311 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:52:01.791367 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:01.791356 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:52:02.291922 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:02.291819 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" event={"ID":"00381713-efff-4f16-8fc8-af879b07e517","Type":"ContainerStarted","Data":"466d89db26928b860cbaf6d4ac0ae763c7f02f4bac4a1318cf781dd6ed487ff5"} Nov 01 22:52:02.292093 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:02.291966 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" event={"ID":"00381713-efff-4f16-8fc8-af879b07e517","Type":"ContainerStarted","Data":"1ba7fca4cf6bbba41b5676b7624d04b8a0a01a310b0161d002fe8229db9c34e5"} Nov 01 22:52:02.296464 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:02.296431 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" event={"ID":"831636af-90aa-41a1-b03f-561c0462ab4e","Type":"ContainerStarted","Data":"ce03038a96f765ded4e8c635d32dab68dd04dd499bcd3402698c9588b828b4bc"} Nov 01 22:52:02.296622 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:02.296486 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" event={"ID":"831636af-90aa-41a1-b03f-561c0462ab4e","Type":"ContainerStarted","Data":"1f980635acef8d1917619003d75a21524a5c8836518cf06cc14d15b857574d89"} Nov 01 22:52:02.300517 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:02.300387 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" event={"ID":"ee08f7f3-27a6-44fc-9099-dbe8459e6b12","Type":"ContainerStarted","Data":"f16d573efb1b8b90ba0416907e229aed0beb62e88fa7f0546b2f83b48ed7e0ed"} Nov 01 22:52:02.300517 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:02.300446 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" event={"ID":"ee08f7f3-27a6-44fc-9099-dbe8459e6b12","Type":"ContainerStarted","Data":"9f642835dfb492dae66015066b3d1e008807919cd26b73511d145da33f2c8a95"} Nov 01 22:52:02.325119 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:02.325053 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" podStartSLOduration=84.325033995 podStartE2EDuration="1m24.325033995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:52:02.317777542 +0000 UTC m=+1690.722368175" watchObservedRunningTime="2026-11-01 22:52:02.325033995 +0000 UTC m=+1690.729624628" Nov 01 22:52:02.347859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:02.342862 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" podStartSLOduration=84.342823623 podStartE2EDuration="1m24.342823623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:52:02.340376759 +0000 UTC m=+1690.744967393" watchObservedRunningTime="2026-11-01 22:52:02.342823623 +0000 UTC m=+1690.747414258" Nov 01 22:52:02.436343 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:02.436311 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:02.439865 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:02.439814 5011 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 01 22:52:02.451506 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:02.451458 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" podUID="00381713-efff-4f16-8fc8-af879b07e517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 01 22:52:03.036780 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:03.034546 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:03.037458 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:03.037399 5011 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 01 22:52:03.037568 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:03.037460 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" podUID="831636af-90aa-41a1-b03f-561c0462ab4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 01 22:52:03.118326 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:03.116885 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:03.122020 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:03.121637 5011 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 01 22:52:03.122020 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:03.121690 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" podUID="ee08f7f3-27a6-44fc-9099-dbe8459e6b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 01 22:52:03.426796 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:03.426318 5011 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 01 22:52:03.426796 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:03.426381 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" podUID="00381713-efff-4f16-8fc8-af879b07e517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 01 22:52:04.030730 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:04.030697 5011 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 01 22:52:04.031164 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:04.031126 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" podUID="831636af-90aa-41a1-b03f-561c0462ab4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 01 22:52:04.121252 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:04.121216 5011 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 01 22:52:04.122128 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:04.122093 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" podUID="ee08f7f3-27a6-44fc-9099-dbe8459e6b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 01 22:52:04.444910 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:04.441159 5011 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 01 22:52:04.444910 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:04.441235 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" podUID="00381713-efff-4f16-8fc8-af879b07e517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 01 22:52:05.035644 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:05.032397 5011 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 01 22:52:05.035644 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:05.032521 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" podUID="831636af-90aa-41a1-b03f-561c0462ab4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 01 22:52:05.119869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:05.119227 5011 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 01 22:52:05.119869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:05.119289 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" podUID="ee08f7f3-27a6-44fc-9099-dbe8459e6b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 01 22:52:05.431861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:05.427236 5011 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 01 22:52:05.431861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:05.427314 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" podUID="00381713-efff-4f16-8fc8-af879b07e517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 01 22:52:06.032318 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:06.031557 5011 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 01 22:52:06.032318 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:06.031616 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" podUID="831636af-90aa-41a1-b03f-561c0462ab4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 01 22:52:06.119861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:06.118429 5011 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 01 22:52:06.119861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:06.118547 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" podUID="ee08f7f3-27a6-44fc-9099-dbe8459e6b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 01 22:52:06.426929 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:06.426500 5011 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 01 22:52:06.426929 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:06.426577 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" podUID="00381713-efff-4f16-8fc8-af879b07e517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 01 22:52:07.036876 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:07.036070 5011 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 01 22:52:07.036876 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:07.036191 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" podUID="831636af-90aa-41a1-b03f-561c0462ab4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 01 22:52:07.117276 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:07.117239 5011 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 01 22:52:07.117573 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:07.117543 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" podUID="ee08f7f3-27a6-44fc-9099-dbe8459e6b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 01 22:52:07.426870 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:07.426003 5011 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 01 22:52:07.426870 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:07.426065 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" podUID="00381713-efff-4f16-8fc8-af879b07e517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 01 22:52:08.032911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:08.031427 5011 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 01 22:52:08.032911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:08.031482 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" podUID="831636af-90aa-41a1-b03f-561c0462ab4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 01 22:52:08.128859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:08.124305 5011 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 01 22:52:08.128859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:08.124370 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" podUID="ee08f7f3-27a6-44fc-9099-dbe8459e6b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 01 22:52:08.427456 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:08.427410 5011 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 01 22:52:08.427664 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:08.427468 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" podUID="00381713-efff-4f16-8fc8-af879b07e517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 01 22:52:09.117987 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:09.117957 5011 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 01 22:52:09.118224 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:09.118202 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" podUID="ee08f7f3-27a6-44fc-9099-dbe8459e6b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 01 22:52:09.273921 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:09.273413 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:09.279870 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:09.275001 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:09.279870 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:09.276410 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:52:09.299392 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:09.299326 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" podStartSLOduration=91.299307638 podStartE2EDuration="1m31.299307638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:52:02.372189082 +0000 UTC m=+1690.776779715" watchObservedRunningTime="2026-11-01 22:52:09.299307638 +0000 UTC m=+1697.703898271" Nov 01 22:52:09.426291 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:09.426220 5011 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 01 22:52:09.426291 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:09.426284 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" podUID="00381713-efff-4f16-8fc8-af879b07e517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 01 22:52:10.123027 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:10.122982 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:10.124432 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:10.124410 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:10.125117 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:10.125101 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:52:10.433279 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:10.433245 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:10.433689 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:10.433664 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:10.434373 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:52:10.434353 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:52:16.962270 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:52:16.962226 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:52:46.962777 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:52:46.962731 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:53:04.894650 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:53:04.894612 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:53:16.968543 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:53:16.968493 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:53:46.963767 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:53:46.963720 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:54:16.961776 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:54:16.961693 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:54:43.380405 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:43.380367 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:54:43.846822 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:43.846786 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1799f8-fb9c976d7-6kds5"] Nov 01 22:54:46.965417 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:54:46.965372 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:54:48.847864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:48.847598 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1799f8-fb9c976d7-6kds5" podUID="389a14f5-f03c-494c-8f3c-7490c443ad6c" containerName="payload-pre-processing" containerID="cri-o://89ea92eccc0c30cfb5acb2afaad343781ab9cab2d635660cb5b650b03029b274" gracePeriod=25 Nov 01 22:54:49.258916 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:49.258881 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm"] Nov 01 22:54:49.259164 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:49.259138 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm" podUID="b6565ccc-c614-436a-8029-9e343d9ff296" containerName="maas-api" containerID="cri-o://efe77cb92b575a47fc26c58554c54fdac89e4f6d7914292eccf01d1f3012d04d" gracePeriod=30 Nov 01 22:54:49.795557 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:49.794818 5011 generic.go:356] "Generic (PLEG): container finished" podID="b6565ccc-c614-436a-8029-9e343d9ff296" containerID="efe77cb92b575a47fc26c58554c54fdac89e4f6d7914292eccf01d1f3012d04d" exitCode=0 Nov 01 22:54:49.795557 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:49.795056 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm" event={"ID":"b6565ccc-c614-436a-8029-9e343d9ff296","Type":"ContainerDied","Data":"efe77cb92b575a47fc26c58554c54fdac89e4f6d7914292eccf01d1f3012d04d"} Nov 01 22:54:50.094390 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.094359 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm" Nov 01 22:54:50.124493 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.124412 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht"] Nov 01 22:54:50.125395 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.125368 5011 cpu_manager.go:414] "containerMap: removing container" podUID="b6565ccc-c614-436a-8029-9e343d9ff296" containerName="maas-api" Nov 01 22:54:50.125395 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.125387 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6565ccc-c614-436a-8029-9e343d9ff296" containerName="maas-api" Nov 01 22:54:50.125550 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.125518 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6565ccc-c614-436a-8029-9e343d9ff296" containerName="maas-api" Nov 01 22:54:50.129501 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.129483 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:50.137492 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.137464 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-44a9aa-openshift-default-dockercfg-h6v7m\"" Nov 01 22:54:50.150074 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.149517 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht"] Nov 01 22:54:50.279826 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.279789 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6565ccc-c614-436a-8029-9e343d9ff296-kube-api-access-mnzf4\" (UniqueName: \"kubernetes.io/projected/b6565ccc-c614-436a-8029-9e343d9ff296-kube-api-access-mnzf4\") pod \"b6565ccc-c614-436a-8029-9e343d9ff296\" (UID: \"b6565ccc-c614-436a-8029-9e343d9ff296\") " Nov 01 22:54:50.280041 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.279892 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6565ccc-c614-436a-8029-9e343d9ff296-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6565ccc-c614-436a-8029-9e343d9ff296-maas-api-tls\") pod \"b6565ccc-c614-436a-8029-9e343d9ff296\" (UID: \"b6565ccc-c614-436a-8029-9e343d9ff296\") " Nov 01 22:54:50.280041 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.279952 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6565ccc-c614-436a-8029-9e343d9ff296-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6565ccc-c614-436a-8029-9e343d9ff296-openshift-service-ca\") pod \"b6565ccc-c614-436a-8029-9e343d9ff296\" (UID: \"b6565ccc-c614-436a-8029-9e343d9ff296\") " Nov 01 22:54:50.280147 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.280130 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6565ccc-c614-436a-8029-9e343d9ff296-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6565ccc-c614-436a-8029-9e343d9ff296-metrics-certs\") pod \"b6565ccc-c614-436a-8029-9e343d9ff296\" (UID: \"b6565ccc-c614-436a-8029-9e343d9ff296\") " Nov 01 22:54:50.280403 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.280382 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6565ccc-c614-436a-8029-9e343d9ff296-openshift-service-ca" pod "b6565ccc-c614-436a-8029-9e343d9ff296" (UID: "b6565ccc-c614-436a-8029-9e343d9ff296"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:54:50.280493 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.280471 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istio-podinfo\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:50.280547 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.280525 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istio-envoy\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:50.280656 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.280629 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-credential-socket\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:50.281191 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.280738 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istio-token\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:50.281191 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.280821 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62m66\" (UniqueName: \"kubernetes.io/projected/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-kube-api-access-62m66\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:50.281191 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.280888 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istiod-ca-cert\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:50.281191 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.281111 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-workload-certs\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:50.281191 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.281171 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istio-data\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:50.281432 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.281304 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-workload-socket\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:50.281707 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.281674 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6565ccc-c614-436a-8029-9e343d9ff296-openshift-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:50.282649 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.282627 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6565ccc-c614-436a-8029-9e343d9ff296-maas-api-tls" pod "b6565ccc-c614-436a-8029-9e343d9ff296" (UID: "b6565ccc-c614-436a-8029-9e343d9ff296"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:54:50.282764 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.282715 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6565ccc-c614-436a-8029-9e343d9ff296-metrics-certs" pod "b6565ccc-c614-436a-8029-9e343d9ff296" (UID: "b6565ccc-c614-436a-8029-9e343d9ff296"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:54:50.282764 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.282740 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6565ccc-c614-436a-8029-9e343d9ff296-kube-api-access-mnzf4" pod "b6565ccc-c614-436a-8029-9e343d9ff296" (UID: "b6565ccc-c614-436a-8029-9e343d9ff296"). InnerVolumeSpecName "kube-api-access-mnzf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:54:50.382820 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.382729 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-workload-certs\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:50.382820 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.382818 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istio-data\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:50.383079 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.382904 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-workload-socket\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:50.383142 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.383094 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istio-podinfo\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:50.383204 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.383138 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istio-envoy\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:50.383204 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.383168 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-credential-socket\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:50.383204 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.383183 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-workload-certs\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:50.383204 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.383197 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istio-token\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:50.383419 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.383251 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62m66\" (UniqueName: \"kubernetes.io/projected/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-kube-api-access-62m66\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:50.383419 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.383288 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istiod-ca-cert\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:50.383419 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.383354 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istio-data\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:50.383582 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.383558 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-credential-socket\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:50.383628 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.383577 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6565ccc-c614-436a-8029-9e343d9ff296-metrics-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:50.383765 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.383740 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnzf4\" (UniqueName: \"kubernetes.io/projected/b6565ccc-c614-436a-8029-9e343d9ff296-kube-api-access-mnzf4\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:50.383949 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.383770 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6565ccc-c614-436a-8029-9e343d9ff296-maas-api-tls\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:50.383949 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.383743 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-workload-socket\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:50.384155 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.384134 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istiod-ca-cert\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:50.386463 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.386441 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istio-envoy\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:50.386650 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.386631 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istio-podinfo\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:50.393233 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.393210 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62m66\" (UniqueName: \"kubernetes.io/projected/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-kube-api-access-62m66\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:50.397196 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.397176 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istio-token\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:50.457257 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.457221 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:50.759313 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.759282 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht"] Nov 01 22:54:50.760116 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:54:50.760090 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f7128f_2e3a_4c22_ad5f_9ee3b219c89d.slice/crio-dcdda9ae4c36e3367a4ffbe6a052b5ece36d1630d91033e250490e2fb491f803 WatchSource:0}: Error finding container dcdda9ae4c36e3367a4ffbe6a052b5ece36d1630d91033e250490e2fb491f803: Status 404 returned error can't find the container with id dcdda9ae4c36e3367a4ffbe6a052b5ece36d1630d91033e250490e2fb491f803 Nov 01 22:54:50.764053 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.764013 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:54:50.764145 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.764111 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:54:50.764202 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.764155 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:54:50.899058 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.898766 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm" event={"ID":"b6565ccc-c614-436a-8029-9e343d9ff296","Type":"ContainerDied","Data":"9589ed5334dc728aa879d75e9ef01500972131d3591ede5cb90eeee918018e04"} Nov 01 22:54:50.899058 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.898911 5011 scope.go:122] "RemoveContainer" containerID="efe77cb92b575a47fc26c58554c54fdac89e4f6d7914292eccf01d1f3012d04d" Nov 01 22:54:50.899058 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.898916 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm" Nov 01 22:54:50.903322 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.903294 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" event={"ID":"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d","Type":"ContainerStarted","Data":"dcdda9ae4c36e3367a4ffbe6a052b5ece36d1630d91033e250490e2fb491f803"} Nov 01 22:54:50.945947 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.945908 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm"] Nov 01 22:54:50.950249 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:50.950219 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1799f8-54bdd4988d-5cvhm"] Nov 01 22:54:51.785888 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:51.785856 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6565ccc-c614-436a-8029-9e343d9ff296" path="/var/lib/kubelet/pods/b6565ccc-c614-436a-8029-9e343d9ff296/volumes" Nov 01 22:54:51.914188 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:51.914154 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" event={"ID":"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d","Type":"ContainerStarted","Data":"010146e0b492f013ded85565cd2e87cc147e1ee0ef7f44fbd29eb721a24a4d48"} Nov 01 22:54:51.940591 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:51.940538 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" podStartSLOduration=35.940524677 podStartE2EDuration="35.940524677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:54:51.937471569 +0000 UTC m=+1860.342062205" watchObservedRunningTime="2026-11-01 22:54:51.940524677 +0000 UTC m=+1860.345115382" Nov 01 22:54:52.463858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:52.462564 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:52.472735 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:52.472699 5011 patch_prober.go:29] interesting pod/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 01 22:54:52.473099 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:52.473053 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" podUID="35f7128f-2e3a-4c22-ad5f-9ee3b219c89d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 01 22:54:53.466352 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:53.466312 5011 patch_prober.go:29] interesting pod/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 01 22:54:53.467010 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:53.466976 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" podUID="35f7128f-2e3a-4c22-ad5f-9ee3b219c89d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 01 22:54:53.501030 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:53.500988 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1799f8-fb9c976d7-6kds5" podUID="389a14f5-f03c-494c-8f3c-7490c443ad6c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:54:54.463856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:54.463048 5011 patch_prober.go:29] interesting pod/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 01 22:54:54.463856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:54.463106 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" podUID="35f7128f-2e3a-4c22-ad5f-9ee3b219c89d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 01 22:54:55.458860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:55.457767 5011 patch_prober.go:29] interesting pod/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 01 22:54:55.458860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:55.457871 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" podUID="35f7128f-2e3a-4c22-ad5f-9ee3b219c89d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 01 22:54:56.467521 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:56.467010 5011 patch_prober.go:29] interesting pod/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 01 22:54:56.467521 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:56.467085 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" podUID="35f7128f-2e3a-4c22-ad5f-9ee3b219c89d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 01 22:54:57.638237 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:57.630780 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:57.638237 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:57.631560 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:57.647960 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:57.644171 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:54:57.865244 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:57.865210 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz"] Nov 01 22:54:57.865437 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:54:57.865394 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" podUID="4923bea7-5f7a-4fb4-a064-d4ffa307379a" containerName="istio-proxy" containerID="cri-o://40f1dfe0d3bcc9c2b39c3bb22ab045fe55bfee32124afd322207f6d04f652bd2" gracePeriod=30 Nov 01 22:55:03.159671 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.159638 5011 generic.go:356] "Generic (PLEG): container finished" podID="4923bea7-5f7a-4fb4-a064-d4ffa307379a" containerID="40f1dfe0d3bcc9c2b39c3bb22ab045fe55bfee32124afd322207f6d04f652bd2" exitCode=0 Nov 01 22:55:03.160079 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.159707 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" event={"ID":"4923bea7-5f7a-4fb4-a064-d4ffa307379a","Type":"ContainerDied","Data":"40f1dfe0d3bcc9c2b39c3bb22ab045fe55bfee32124afd322207f6d04f652bd2"} Nov 01 22:55:03.160079 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.159762 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" event={"ID":"4923bea7-5f7a-4fb4-a064-d4ffa307379a","Type":"ContainerDied","Data":"5125224a6a3da5c61eae4cc6a6c8f3d47c83d4a619482748badae4958192146c"} Nov 01 22:55:03.160079 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.159783 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5125224a6a3da5c61eae4cc6a6c8f3d47c83d4a619482748badae4958192146c" Nov 01 22:55:03.189994 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.189960 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:55:03.226022 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.225988 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-64b196-5c56f9cc87-sqg9q"] Nov 01 22:55:03.227002 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.226979 5011 cpu_manager.go:414] "containerMap: removing container" podUID="4923bea7-5f7a-4fb4-a064-d4ffa307379a" containerName="istio-proxy" Nov 01 22:55:03.227002 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.227000 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4923bea7-5f7a-4fb4-a064-d4ffa307379a" containerName="istio-proxy" Nov 01 22:55:03.227257 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.227242 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="4923bea7-5f7a-4fb4-a064-d4ffa307379a" containerName="istio-proxy" Nov 01 22:55:03.231154 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.231132 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-64b196-5c56f9cc87-sqg9q" Nov 01 22:55:03.239574 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.236692 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-64b196\"" Nov 01 22:55:03.239574 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.236797 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-64b196-dockercfg-fpncx\"" Nov 01 22:55:03.241182 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.239592 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-64b196-5c56f9cc87-sqg9q"] Nov 01 22:55:03.259525 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.259495 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-credential-socket\") pod \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " Nov 01 22:55:03.259677 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.259577 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-workload-certs\") pod \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " Nov 01 22:55:03.259677 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.259642 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istio-token\" (UniqueName: \"kubernetes.io/projected/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istio-token\") pod \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " Nov 01 22:55:03.259807 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.259784 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-credential-socket" pod "4923bea7-5f7a-4fb4-a064-d4ffa307379a" (UID: "4923bea7-5f7a-4fb4-a064-d4ffa307379a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:03.259892 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.259806 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istio-data\") pod \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " Nov 01 22:55:03.259991 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.259973 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istiod-ca-cert\") pod \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " Nov 01 22:55:03.260055 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.260018 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istio-podinfo\") pod \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " Nov 01 22:55:03.260116 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.259955 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-workload-certs" pod "4923bea7-5f7a-4fb4-a064-d4ffa307379a" (UID: "4923bea7-5f7a-4fb4-a064-d4ffa307379a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:03.260116 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.260010 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istio-data" pod "4923bea7-5f7a-4fb4-a064-d4ffa307379a" (UID: "4923bea7-5f7a-4fb4-a064-d4ffa307379a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:03.260225 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.260121 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-workload-socket\") pod \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " Nov 01 22:55:03.260225 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.260183 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4923bea7-5f7a-4fb4-a064-d4ffa307379a-kube-api-access-j7wn7\" (UniqueName: \"kubernetes.io/projected/4923bea7-5f7a-4fb4-a064-d4ffa307379a-kube-api-access-j7wn7\") pod \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " Nov 01 22:55:03.260330 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.260224 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istio-envoy\") pod \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\" (UID: \"4923bea7-5f7a-4fb4-a064-d4ffa307379a\") " Nov 01 22:55:03.260330 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.260266 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istiod-ca-cert" pod "4923bea7-5f7a-4fb4-a064-d4ffa307379a" (UID: "4923bea7-5f7a-4fb4-a064-d4ffa307379a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:55:03.260619 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.260596 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-workload-socket" pod "4923bea7-5f7a-4fb4-a064-d4ffa307379a" (UID: "4923bea7-5f7a-4fb4-a064-d4ffa307379a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:03.261058 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.261035 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:03.261146 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.261063 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:03.261146 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.261079 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:03.261146 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.261092 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:03.261146 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.261105 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:03.263664 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.263391 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istio-envoy" pod "4923bea7-5f7a-4fb4-a064-d4ffa307379a" (UID: "4923bea7-5f7a-4fb4-a064-d4ffa307379a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:03.265130 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.265108 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istio-podinfo" pod "4923bea7-5f7a-4fb4-a064-d4ffa307379a" (UID: "4923bea7-5f7a-4fb4-a064-d4ffa307379a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:55:03.265235 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.265150 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istio-token" pod "4923bea7-5f7a-4fb4-a064-d4ffa307379a" (UID: "4923bea7-5f7a-4fb4-a064-d4ffa307379a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:03.267026 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.266997 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4923bea7-5f7a-4fb4-a064-d4ffa307379a-kube-api-access-j7wn7" pod "4923bea7-5f7a-4fb4-a064-d4ffa307379a" (UID: "4923bea7-5f7a-4fb4-a064-d4ffa307379a"). InnerVolumeSpecName "kube-api-access-j7wn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:03.362751 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.362717 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e42f2aff-b236-41b5-842e-0e4cb93f8b92-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-64b196-5c56f9cc87-sqg9q\" (UID: \"e42f2aff-b236-41b5-842e-0e4cb93f8b92\") " pod="openshift-ingress/payload-processing-e2e-inf-b-64b196-5c56f9cc87-sqg9q" Nov 01 22:55:03.363175 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.363100 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzt57\" (UniqueName: \"kubernetes.io/projected/e42f2aff-b236-41b5-842e-0e4cb93f8b92-kube-api-access-wzt57\") pod \"payload-processing-e2e-inf-b-64b196-5c56f9cc87-sqg9q\" (UID: \"e42f2aff-b236-41b5-842e-0e4cb93f8b92\") " pod="openshift-ingress/payload-processing-e2e-inf-b-64b196-5c56f9cc87-sqg9q" Nov 01 22:55:03.363484 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.363462 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:03.363522 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.363495 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7wn7\" (UniqueName: \"kubernetes.io/projected/4923bea7-5f7a-4fb4-a064-d4ffa307379a-kube-api-access-j7wn7\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:03.363522 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.363513 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:03.363590 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.363529 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4923bea7-5f7a-4fb4-a064-d4ffa307379a-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:03.465522 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.465479 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e42f2aff-b236-41b5-842e-0e4cb93f8b92-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-64b196-5c56f9cc87-sqg9q\" (UID: \"e42f2aff-b236-41b5-842e-0e4cb93f8b92\") " pod="openshift-ingress/payload-processing-e2e-inf-b-64b196-5c56f9cc87-sqg9q" Nov 01 22:55:03.465774 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.465758 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzt57\" (UniqueName: \"kubernetes.io/projected/e42f2aff-b236-41b5-842e-0e4cb93f8b92-kube-api-access-wzt57\") pod \"payload-processing-e2e-inf-b-64b196-5c56f9cc87-sqg9q\" (UID: \"e42f2aff-b236-41b5-842e-0e4cb93f8b92\") " pod="openshift-ingress/payload-processing-e2e-inf-b-64b196-5c56f9cc87-sqg9q" Nov 01 22:55:03.466470 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.466445 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e42f2aff-b236-41b5-842e-0e4cb93f8b92-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-64b196-5c56f9cc87-sqg9q\" (UID: \"e42f2aff-b236-41b5-842e-0e4cb93f8b92\") " pod="openshift-ingress/payload-processing-e2e-inf-b-64b196-5c56f9cc87-sqg9q" Nov 01 22:55:03.474200 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.474177 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzt57\" (UniqueName: \"kubernetes.io/projected/e42f2aff-b236-41b5-842e-0e4cb93f8b92-kube-api-access-wzt57\") pod \"payload-processing-e2e-inf-b-64b196-5c56f9cc87-sqg9q\" (UID: \"e42f2aff-b236-41b5-842e-0e4cb93f8b92\") " pod="openshift-ingress/payload-processing-e2e-inf-b-64b196-5c56f9cc87-sqg9q" Nov 01 22:55:03.501060 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.501027 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1799f8-fb9c976d7-6kds5" podUID="389a14f5-f03c-494c-8f3c-7490c443ad6c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:55:03.559351 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:03.559315 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-64b196-5c56f9cc87-sqg9q" Nov 01 22:55:04.046032 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:04.045997 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-64b196-5c56f9cc87-sqg9q"] Nov 01 22:55:04.046811 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:55:04.046779 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode42f2aff_b236_41b5_842e_0e4cb93f8b92.slice/crio-9d04c093f836aa8e12a6857a76ede898188c26bce765478a49327e2f6d1371af WatchSource:0}: Error finding container 9d04c093f836aa8e12a6857a76ede898188c26bce765478a49327e2f6d1371af: Status 404 returned error can't find the container with id 9d04c093f836aa8e12a6857a76ede898188c26bce765478a49327e2f6d1371af Nov 01 22:55:04.174635 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:04.174587 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-64b196-5c56f9cc87-sqg9q" event={"ID":"e42f2aff-b236-41b5-842e-0e4cb93f8b92","Type":"ContainerStarted","Data":"7cd9b9428bbe35c031ed7caff3c6e7dd6570c2b55ab5bab7999d7831b74b1165"} Nov 01 22:55:04.175195 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:04.174662 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-64b196-5c56f9cc87-sqg9q" event={"ID":"e42f2aff-b236-41b5-842e-0e4cb93f8b92","Type":"ContainerStarted","Data":"9d04c093f836aa8e12a6857a76ede898188c26bce765478a49327e2f6d1371af"} Nov 01 22:55:04.175195 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:04.174702 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz" Nov 01 22:55:04.175195 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:04.174710 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-64b196-5c56f9cc87-sqg9q" Nov 01 22:55:04.199210 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:04.199153 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-64b196-5c56f9cc87-sqg9q" podStartSLOduration=118.199136662 podStartE2EDuration="1m58.199136662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:04.198041853 +0000 UTC m=+1872.602632486" watchObservedRunningTime="2026-11-01 22:55:04.199136662 +0000 UTC m=+1872.603727310" Nov 01 22:55:04.218420 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:04.218391 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz"] Nov 01 22:55:04.224628 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:04.224601 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-f4mhz"] Nov 01 22:55:05.752210 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:05.752171 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4923bea7-5f7a-4fb4-a064-d4ffa307379a" path="/var/lib/kubelet/pods/4923bea7-5f7a-4fb4-a064-d4ffa307379a/volumes" Nov 01 22:55:09.285030 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:09.284991 5011 generic.go:356] "Generic (PLEG): container finished" podID="389a14f5-f03c-494c-8f3c-7490c443ad6c" containerID="89ea92eccc0c30cfb5acb2afaad343781ab9cab2d635660cb5b650b03029b274" exitCode=0 Nov 01 22:55:09.285515 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:09.285066 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1799f8-fb9c976d7-6kds5" event={"ID":"389a14f5-f03c-494c-8f3c-7490c443ad6c","Type":"ContainerDied","Data":"89ea92eccc0c30cfb5acb2afaad343781ab9cab2d635660cb5b650b03029b274"} Nov 01 22:55:09.504849 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:09.504747 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1799f8-fb9c976d7-6kds5" Nov 01 22:55:09.596421 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:09.596383 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/389a14f5-f03c-494c-8f3c-7490c443ad6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/389a14f5-f03c-494c-8f3c-7490c443ad6c-plugins-config-volume\") pod \"389a14f5-f03c-494c-8f3c-7490c443ad6c\" (UID: \"389a14f5-f03c-494c-8f3c-7490c443ad6c\") " Nov 01 22:55:09.596743 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:09.596720 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/389a14f5-f03c-494c-8f3c-7490c443ad6c-kube-api-access-rpbks\" (UniqueName: \"kubernetes.io/projected/389a14f5-f03c-494c-8f3c-7490c443ad6c-kube-api-access-rpbks\") pod \"389a14f5-f03c-494c-8f3c-7490c443ad6c\" (UID: \"389a14f5-f03c-494c-8f3c-7490c443ad6c\") " Nov 01 22:55:09.600353 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:09.600323 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/389a14f5-f03c-494c-8f3c-7490c443ad6c-plugins-config-volume" pod "389a14f5-f03c-494c-8f3c-7490c443ad6c" (UID: "389a14f5-f03c-494c-8f3c-7490c443ad6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:55:09.602073 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:09.602047 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/389a14f5-f03c-494c-8f3c-7490c443ad6c-kube-api-access-rpbks" pod "389a14f5-f03c-494c-8f3c-7490c443ad6c" (UID: "389a14f5-f03c-494c-8f3c-7490c443ad6c"). InnerVolumeSpecName "kube-api-access-rpbks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:09.700460 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:09.700417 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/389a14f5-f03c-494c-8f3c-7490c443ad6c-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:09.700460 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:09.700452 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpbks\" (UniqueName: \"kubernetes.io/projected/389a14f5-f03c-494c-8f3c-7490c443ad6c-kube-api-access-rpbks\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:10.294527 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:10.294484 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1799f8-fb9c976d7-6kds5" event={"ID":"389a14f5-f03c-494c-8f3c-7490c443ad6c","Type":"ContainerDied","Data":"04b0a4bab1a976c9a61ee6033b88d3f7aeb9041169fce173026e7f45491b116b"} Nov 01 22:55:10.294527 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:10.294523 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1799f8-fb9c976d7-6kds5" Nov 01 22:55:10.295081 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:10.294546 5011 scope.go:122] "RemoveContainer" containerID="89ea92eccc0c30cfb5acb2afaad343781ab9cab2d635660cb5b650b03029b274" Nov 01 22:55:10.316673 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:10.316645 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1799f8-fb9c976d7-6kds5"] Nov 01 22:55:10.320895 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:10.320863 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1799f8-fb9c976d7-6kds5"] Nov 01 22:55:11.811355 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:11.811308 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="389a14f5-f03c-494c-8f3c-7490c443ad6c" path="/var/lib/kubelet/pods/389a14f5-f03c-494c-8f3c-7490c443ad6c/volumes" Nov 01 22:55:15.222313 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:15.222280 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-64b196-5c56f9cc87-sqg9q" Nov 01 22:55:16.708225 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:16.708176 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx"] Nov 01 22:55:16.709069 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:16.709011 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx" podUID="1059eeab-0cdf-47fe-997c-71245e8407a2" containerName="maas-api" containerID="cri-o://3ceb6287b6c768d31bd814e82188b136ebabd61a5e2e14ca79de4a111348fdc8" gracePeriod=30 Nov 01 22:55:17.465620 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.465516 5011 generic.go:356] "Generic (PLEG): container finished" podID="1059eeab-0cdf-47fe-997c-71245e8407a2" containerID="3ceb6287b6c768d31bd814e82188b136ebabd61a5e2e14ca79de4a111348fdc8" exitCode=0 Nov 01 22:55:17.465620 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.465574 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx" event={"ID":"1059eeab-0cdf-47fe-997c-71245e8407a2","Type":"ContainerDied","Data":"3ceb6287b6c768d31bd814e82188b136ebabd61a5e2e14ca79de4a111348fdc8"} Nov 01 22:55:17.483111 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:55:17.481473 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:55:17.513012 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.512982 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx" Nov 01 22:55:17.555947 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.555916 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1059eeab-0cdf-47fe-997c-71245e8407a2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1059eeab-0cdf-47fe-997c-71245e8407a2-maas-api-tls\") pod \"1059eeab-0cdf-47fe-997c-71245e8407a2\" (UID: \"1059eeab-0cdf-47fe-997c-71245e8407a2\") " Nov 01 22:55:17.557041 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.557008 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1059eeab-0cdf-47fe-997c-71245e8407a2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1059eeab-0cdf-47fe-997c-71245e8407a2-metrics-certs\") pod \"1059eeab-0cdf-47fe-997c-71245e8407a2\" (UID: \"1059eeab-0cdf-47fe-997c-71245e8407a2\") " Nov 01 22:55:17.557199 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.557096 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1059eeab-0cdf-47fe-997c-71245e8407a2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1059eeab-0cdf-47fe-997c-71245e8407a2-openshift-service-ca\") pod \"1059eeab-0cdf-47fe-997c-71245e8407a2\" (UID: \"1059eeab-0cdf-47fe-997c-71245e8407a2\") " Nov 01 22:55:17.557199 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.557168 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1059eeab-0cdf-47fe-997c-71245e8407a2-kube-api-access-nqmb6\" (UniqueName: \"kubernetes.io/projected/1059eeab-0cdf-47fe-997c-71245e8407a2-kube-api-access-nqmb6\") pod \"1059eeab-0cdf-47fe-997c-71245e8407a2\" (UID: \"1059eeab-0cdf-47fe-997c-71245e8407a2\") " Nov 01 22:55:17.557586 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.557550 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1059eeab-0cdf-47fe-997c-71245e8407a2-openshift-service-ca" pod "1059eeab-0cdf-47fe-997c-71245e8407a2" (UID: "1059eeab-0cdf-47fe-997c-71245e8407a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:55:17.558198 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.558091 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1059eeab-0cdf-47fe-997c-71245e8407a2-openshift-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:17.558754 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.558728 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs"] Nov 01 22:55:17.558890 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.558822 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1059eeab-0cdf-47fe-997c-71245e8407a2-maas-api-tls" pod "1059eeab-0cdf-47fe-997c-71245e8407a2" (UID: "1059eeab-0cdf-47fe-997c-71245e8407a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:55:17.559591 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.559552 5011 cpu_manager.go:414] "containerMap: removing container" podUID="389a14f5-f03c-494c-8f3c-7490c443ad6c" containerName="payload-pre-processing" Nov 01 22:55:17.559646 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.559594 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="389a14f5-f03c-494c-8f3c-7490c443ad6c" containerName="payload-pre-processing" Nov 01 22:55:17.559703 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.559654 5011 cpu_manager.go:414] "containerMap: removing container" podUID="1059eeab-0cdf-47fe-997c-71245e8407a2" containerName="maas-api" Nov 01 22:55:17.559703 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.559660 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1059eeab-0cdf-47fe-997c-71245e8407a2" containerName="maas-api" Nov 01 22:55:17.559818 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.559788 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="389a14f5-f03c-494c-8f3c-7490c443ad6c" containerName="payload-pre-processing" Nov 01 22:55:17.559818 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.559807 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="1059eeab-0cdf-47fe-997c-71245e8407a2" containerName="maas-api" Nov 01 22:55:17.560080 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.560059 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1059eeab-0cdf-47fe-997c-71245e8407a2-kube-api-access-nqmb6" pod "1059eeab-0cdf-47fe-997c-71245e8407a2" (UID: "1059eeab-0cdf-47fe-997c-71245e8407a2"). InnerVolumeSpecName "kube-api-access-nqmb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:17.560140 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.560114 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1059eeab-0cdf-47fe-997c-71245e8407a2-metrics-certs" pod "1059eeab-0cdf-47fe-997c-71245e8407a2" (UID: "1059eeab-0cdf-47fe-997c-71245e8407a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:55:17.588731 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.588691 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs"] Nov 01 22:55:17.588955 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.588885 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:17.592626 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.592597 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b48ab8-openshift-default-dockercfg-2r4pn\"" Nov 01 22:55:17.667170 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.667134 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89a5b17f-1549-470c-8968-01b013df9417-istio-podinfo\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:17.667418 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.667214 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-istio-envoy\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:17.667418 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.667363 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-workload-certs\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:17.667418 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.667409 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89a5b17f-1549-470c-8968-01b013df9417-istiod-ca-cert\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:17.667613 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.667509 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-workload-socket\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:17.667613 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.667531 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-istio-data\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:17.667613 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.667596 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9nmg\" (UniqueName: \"kubernetes.io/projected/89a5b17f-1549-470c-8968-01b013df9417-kube-api-access-r9nmg\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:17.667769 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.667652 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89a5b17f-1549-470c-8968-01b013df9417-istio-token\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:17.667769 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.667692 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-credential-socket\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:17.668227 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.668190 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqmb6\" (UniqueName: \"kubernetes.io/projected/1059eeab-0cdf-47fe-997c-71245e8407a2-kube-api-access-nqmb6\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:17.668227 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.668220 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1059eeab-0cdf-47fe-997c-71245e8407a2-maas-api-tls\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:17.668399 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.668235 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1059eeab-0cdf-47fe-997c-71245e8407a2-metrics-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:17.769453 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.769423 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-credential-socket\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:17.769908 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.769500 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89a5b17f-1549-470c-8968-01b013df9417-istio-podinfo\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:17.769908 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.769539 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-istio-envoy\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:17.769908 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.769658 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-workload-certs\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:17.769908 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.769725 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89a5b17f-1549-470c-8968-01b013df9417-istiod-ca-cert\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:17.769908 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.769791 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-workload-socket\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:17.769908 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.769821 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-istio-data\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:17.770274 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.769905 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-credential-socket\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:17.770274 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.769929 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9nmg\" (UniqueName: \"kubernetes.io/projected/89a5b17f-1549-470c-8968-01b013df9417-kube-api-access-r9nmg\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:17.770274 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.770017 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89a5b17f-1549-470c-8968-01b013df9417-istio-token\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:17.770274 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.770164 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-workload-certs\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:17.770274 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.770219 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-workload-socket\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:17.770274 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.770261 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-istio-data\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:17.770757 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.770736 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89a5b17f-1549-470c-8968-01b013df9417-istiod-ca-cert\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:17.772049 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.772025 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-istio-envoy\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:17.772698 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.772643 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89a5b17f-1549-470c-8968-01b013df9417-istio-podinfo\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:17.778633 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.778611 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89a5b17f-1549-470c-8968-01b013df9417-istio-token\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:17.779914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.779896 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9nmg\" (UniqueName: \"kubernetes.io/projected/89a5b17f-1549-470c-8968-01b013df9417-kube-api-access-r9nmg\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:17.907726 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:17.907695 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:18.208381 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:55:18.208347 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89a5b17f_1549_470c_8968_01b013df9417.slice/crio-ffd30464ba6874316fb19891a12058938ab3a6002c82f75c0eef63f6a0a04114 WatchSource:0}: Error finding container ffd30464ba6874316fb19891a12058938ab3a6002c82f75c0eef63f6a0a04114: Status 404 returned error can't find the container with id ffd30464ba6874316fb19891a12058938ab3a6002c82f75c0eef63f6a0a04114 Nov 01 22:55:18.209049 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:18.209020 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs"] Nov 01 22:55:18.212171 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:18.212131 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:55:18.212309 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:18.212219 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:55:18.212309 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:18.212260 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:55:18.497482 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:18.497383 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx" event={"ID":"1059eeab-0cdf-47fe-997c-71245e8407a2","Type":"ContainerDied","Data":"bf363baa578eef9cbb568664a10085cf092e2d1081728d1dd8de5f1a8f82d5e6"} Nov 01 22:55:18.497703 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:18.497484 5011 scope.go:122] "RemoveContainer" containerID="3ceb6287b6c768d31bd814e82188b136ebabd61a5e2e14ca79de4a111348fdc8" Nov 01 22:55:18.497703 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:18.497400 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx" Nov 01 22:55:18.503329 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:18.503289 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" event={"ID":"89a5b17f-1549-470c-8968-01b013df9417","Type":"ContainerStarted","Data":"92768429b3f0b69587257e3da0d606a65e8b74fcfb37042a5aee1152e0dabe3e"} Nov 01 22:55:18.503468 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:18.503357 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" event={"ID":"89a5b17f-1549-470c-8968-01b013df9417","Type":"ContainerStarted","Data":"ffd30464ba6874316fb19891a12058938ab3a6002c82f75c0eef63f6a0a04114"} Nov 01 22:55:18.521216 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:18.521183 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx"] Nov 01 22:55:18.527100 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:18.527076 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-44a9aa-b8f954687-jxxcx"] Nov 01 22:55:18.544955 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:18.544913 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" podStartSLOduration=67.544898384 podStartE2EDuration="1m7.544898384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:18.542722193 +0000 UTC m=+1886.947312826" watchObservedRunningTime="2026-11-01 22:55:18.544898384 +0000 UTC m=+1886.949489016" Nov 01 22:55:18.908753 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:18.908672 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:18.978262 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:18.978226 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv"] Nov 01 22:55:18.978450 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:18.978424 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" podUID="1b6bec10-9ac2-4708-9631-b76c801c0c58" containerName="istio-proxy" containerID="cri-o://f671104c94e86d233733c47eb490d4a87cdc9694df3d08517c8051a0d9f1270c" gracePeriod=30 Nov 01 22:55:19.749814 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:19.749781 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1059eeab-0cdf-47fe-997c-71245e8407a2" path="/var/lib/kubelet/pods/1059eeab-0cdf-47fe-997c-71245e8407a2/volumes" Nov 01 22:55:19.915286 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:19.915255 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:20.601859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:20.598968 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:20.640874 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:20.638201 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:55:24.472611 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.472585 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:55:24.512245 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.512209 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f"] Nov 01 22:55:24.513847 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.513787 5011 cpu_manager.go:414] "containerMap: removing container" podUID="1b6bec10-9ac2-4708-9631-b76c801c0c58" containerName="istio-proxy" Nov 01 22:55:24.513847 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.513821 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b6bec10-9ac2-4708-9631-b76c801c0c58" containerName="istio-proxy" Nov 01 22:55:24.514128 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.514110 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b6bec10-9ac2-4708-9631-b76c801c0c58" containerName="istio-proxy" Nov 01 22:55:24.523948 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.523925 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:24.530503 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.530464 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-021714-openshift-default-dockercfg-vmtmc\"" Nov 01 22:55:24.538915 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.538846 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f"] Nov 01 22:55:24.566526 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.566493 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-istio-envoy\") pod \"1b6bec10-9ac2-4708-9631-b76c801c0c58\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " Nov 01 22:55:24.566712 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.566549 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-workload-certs\") pod \"1b6bec10-9ac2-4708-9631-b76c801c0c58\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " Nov 01 22:55:24.566712 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.566660 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b6bec10-9ac2-4708-9631-b76c801c0c58-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b6bec10-9ac2-4708-9631-b76c801c0c58-istio-podinfo\") pod \"1b6bec10-9ac2-4708-9631-b76c801c0c58\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " Nov 01 22:55:24.566890 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.566715 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-credential-socket\") pod \"1b6bec10-9ac2-4708-9631-b76c801c0c58\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " Nov 01 22:55:24.566890 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.566780 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b6bec10-9ac2-4708-9631-b76c801c0c58-kube-api-access-p548n\" (UniqueName: \"kubernetes.io/projected/1b6bec10-9ac2-4708-9631-b76c801c0c58-kube-api-access-p548n\") pod \"1b6bec10-9ac2-4708-9631-b76c801c0c58\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " Nov 01 22:55:24.566890 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.566879 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-workload-certs" pod "1b6bec10-9ac2-4708-9631-b76c801c0c58" (UID: "1b6bec10-9ac2-4708-9631-b76c801c0c58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:24.567071 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.567046 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-istio-data\") pod \"1b6bec10-9ac2-4708-9631-b76c801c0c58\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " Nov 01 22:55:24.567131 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.567100 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b6bec10-9ac2-4708-9631-b76c801c0c58-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b6bec10-9ac2-4708-9631-b76c801c0c58-istiod-ca-cert\") pod \"1b6bec10-9ac2-4708-9631-b76c801c0c58\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " Nov 01 22:55:24.567193 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.567154 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b6bec10-9ac2-4708-9631-b76c801c0c58-istio-token\" (UniqueName: \"kubernetes.io/projected/1b6bec10-9ac2-4708-9631-b76c801c0c58-istio-token\") pod \"1b6bec10-9ac2-4708-9631-b76c801c0c58\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " Nov 01 22:55:24.567193 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.567158 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-credential-socket" pod "1b6bec10-9ac2-4708-9631-b76c801c0c58" (UID: "1b6bec10-9ac2-4708-9631-b76c801c0c58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:24.567328 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.567302 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-workload-socket\") pod \"1b6bec10-9ac2-4708-9631-b76c801c0c58\" (UID: \"1b6bec10-9ac2-4708-9631-b76c801c0c58\") " Nov 01 22:55:24.568107 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.567510 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-istio-data" pod "1b6bec10-9ac2-4708-9631-b76c801c0c58" (UID: "1b6bec10-9ac2-4708-9631-b76c801c0c58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:24.568107 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.567610 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-workload-certs\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:24.568107 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.567676 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/665520e7-d3f3-4dd9-a7a5-3992fd827544-istiod-ca-cert\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:24.568107 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.567719 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/665520e7-d3f3-4dd9-a7a5-3992fd827544-istio-podinfo\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:24.568107 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.567782 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-credential-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:24.568107 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.567789 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b6bec10-9ac2-4708-9631-b76c801c0c58-istiod-ca-cert" pod "1b6bec10-9ac2-4708-9631-b76c801c0c58" (UID: "1b6bec10-9ac2-4708-9631-b76c801c0c58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:55:24.568107 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.567807 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/665520e7-d3f3-4dd9-a7a5-3992fd827544-istio-token\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:24.568107 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.568072 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-workload-socket" pod "1b6bec10-9ac2-4708-9631-b76c801c0c58" (UID: "1b6bec10-9ac2-4708-9631-b76c801c0c58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:24.568524 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.568281 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-istio-data\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:24.568524 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.568422 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-workload-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:24.568524 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.568450 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-istio-envoy\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:24.568524 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.568478 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glgwm\" (UniqueName: \"kubernetes.io/projected/665520e7-d3f3-4dd9-a7a5-3992fd827544-kube-api-access-glgwm\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:24.568778 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.568759 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:24.568862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.568781 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b6bec10-9ac2-4708-9631-b76c801c0c58-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:24.568862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.568793 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:24.568862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.568803 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:24.568862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.568811 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:24.570017 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.569972 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-istio-envoy" pod "1b6bec10-9ac2-4708-9631-b76c801c0c58" (UID: "1b6bec10-9ac2-4708-9631-b76c801c0c58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:24.575237 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.575171 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b6bec10-9ac2-4708-9631-b76c801c0c58-istio-podinfo" pod "1b6bec10-9ac2-4708-9631-b76c801c0c58" (UID: "1b6bec10-9ac2-4708-9631-b76c801c0c58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:55:24.575867 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.575825 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b6bec10-9ac2-4708-9631-b76c801c0c58-istio-token" pod "1b6bec10-9ac2-4708-9631-b76c801c0c58" (UID: "1b6bec10-9ac2-4708-9631-b76c801c0c58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:24.576111 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.576086 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b6bec10-9ac2-4708-9631-b76c801c0c58-kube-api-access-p548n" pod "1b6bec10-9ac2-4708-9631-b76c801c0c58" (UID: "1b6bec10-9ac2-4708-9631-b76c801c0c58"). InnerVolumeSpecName "kube-api-access-p548n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:24.618606 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.618567 5011 generic.go:356] "Generic (PLEG): container finished" podID="1b6bec10-9ac2-4708-9631-b76c801c0c58" containerID="f671104c94e86d233733c47eb490d4a87cdc9694df3d08517c8051a0d9f1270c" exitCode=0 Nov 01 22:55:24.618745 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.618633 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" Nov 01 22:55:24.618745 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.618652 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" event={"ID":"1b6bec10-9ac2-4708-9631-b76c801c0c58","Type":"ContainerDied","Data":"f671104c94e86d233733c47eb490d4a87cdc9694df3d08517c8051a0d9f1270c"} Nov 01 22:55:24.618745 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.618724 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv" event={"ID":"1b6bec10-9ac2-4708-9631-b76c801c0c58","Type":"ContainerDied","Data":"e0de9286007eba922827375bbad377993693b44f483a356c500c885c6eb8e431"} Nov 01 22:55:24.619050 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.618752 5011 scope.go:122] "RemoveContainer" containerID="f671104c94e86d233733c47eb490d4a87cdc9694df3d08517c8051a0d9f1270c" Nov 01 22:55:24.638929 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.638908 5011 scope.go:122] "RemoveContainer" containerID="f671104c94e86d233733c47eb490d4a87cdc9694df3d08517c8051a0d9f1270c" Nov 01 22:55:24.639211 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:55:24.639194 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f671104c94e86d233733c47eb490d4a87cdc9694df3d08517c8051a0d9f1270c\": container with ID starting with f671104c94e86d233733c47eb490d4a87cdc9694df3d08517c8051a0d9f1270c not found: ID does not exist" containerID="f671104c94e86d233733c47eb490d4a87cdc9694df3d08517c8051a0d9f1270c" Nov 01 22:55:24.639277 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.639220 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f671104c94e86d233733c47eb490d4a87cdc9694df3d08517c8051a0d9f1270c"} err="failed to get container status \"f671104c94e86d233733c47eb490d4a87cdc9694df3d08517c8051a0d9f1270c\": rpc error: code = NotFound desc = could not find container \"f671104c94e86d233733c47eb490d4a87cdc9694df3d08517c8051a0d9f1270c\": container with ID starting with f671104c94e86d233733c47eb490d4a87cdc9694df3d08517c8051a0d9f1270c not found: ID does not exist" Nov 01 22:55:24.651776 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.651724 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv"] Nov 01 22:55:24.657455 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.657428 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jkvvv"] Nov 01 22:55:24.670478 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.670447 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/665520e7-d3f3-4dd9-a7a5-3992fd827544-istiod-ca-cert\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:24.670655 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.670497 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/665520e7-d3f3-4dd9-a7a5-3992fd827544-istio-podinfo\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:24.670655 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.670619 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-credential-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:24.670655 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.670651 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/665520e7-d3f3-4dd9-a7a5-3992fd827544-istio-token\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:24.670917 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.670895 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-istio-data\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:24.671047 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.671018 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-workload-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:24.671047 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.671039 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-credential-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:24.671123 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.671060 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-istio-envoy\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:24.671123 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.671104 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glgwm\" (UniqueName: \"kubernetes.io/projected/665520e7-d3f3-4dd9-a7a5-3992fd827544-kube-api-access-glgwm\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:24.671225 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.671192 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/665520e7-d3f3-4dd9-a7a5-3992fd827544-istiod-ca-cert\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:24.671293 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.671277 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-workload-certs\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:24.671359 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.671342 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-workload-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:24.671423 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.671409 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b6bec10-9ac2-4708-9631-b76c801c0c58-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:24.671524 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.671425 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b6bec10-9ac2-4708-9631-b76c801c0c58-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:24.671524 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.671435 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b6bec10-9ac2-4708-9631-b76c801c0c58-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:24.671524 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.671445 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p548n\" (UniqueName: \"kubernetes.io/projected/1b6bec10-9ac2-4708-9631-b76c801c0c58-kube-api-access-p548n\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:24.671908 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.671818 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-workload-certs\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:24.672053 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.672032 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-istio-data\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:24.673508 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.673484 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-istio-envoy\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:24.673778 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.673762 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/665520e7-d3f3-4dd9-a7a5-3992fd827544-istio-podinfo\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:24.678796 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.678772 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/665520e7-d3f3-4dd9-a7a5-3992fd827544-istio-token\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:24.678970 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.678952 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glgwm\" (UniqueName: \"kubernetes.io/projected/665520e7-d3f3-4dd9-a7a5-3992fd827544-kube-api-access-glgwm\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:24.868554 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:24.868469 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:25.166004 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:55:25.165971 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod665520e7_d3f3_4dd9_a7a5_3992fd827544.slice/crio-101415afcfcc565e264a83897c47659a472836fcd379f8b586ab08d1206e3601 WatchSource:0}: Error finding container 101415afcfcc565e264a83897c47659a472836fcd379f8b586ab08d1206e3601: Status 404 returned error can't find the container with id 101415afcfcc565e264a83897c47659a472836fcd379f8b586ab08d1206e3601 Nov 01 22:55:25.166301 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:25.166275 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f"] Nov 01 22:55:25.169402 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:25.169367 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:55:25.169521 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:25.169445 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:55:25.169521 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:25.169494 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:55:25.630732 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:25.630696 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" event={"ID":"665520e7-d3f3-4dd9-a7a5-3992fd827544","Type":"ContainerStarted","Data":"dd45e515bc228815d2ab2ea472d08728c59cdb6b2bb7dd947fb3e9ac0a2414bd"} Nov 01 22:55:25.631174 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:25.630765 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" event={"ID":"665520e7-d3f3-4dd9-a7a5-3992fd827544","Type":"ContainerStarted","Data":"101415afcfcc565e264a83897c47659a472836fcd379f8b586ab08d1206e3601"} Nov 01 22:55:25.657642 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:25.657563 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" podStartSLOduration=74.657535101 podStartE2EDuration="1m14.657535101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:25.652571642 +0000 UTC m=+1894.057162274" watchObservedRunningTime="2026-11-01 22:55:25.657535101 +0000 UTC m=+1894.062125735" Nov 01 22:55:25.750552 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:25.750519 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b6bec10-9ac2-4708-9631-b76c801c0c58" path="/var/lib/kubelet/pods/1b6bec10-9ac2-4708-9631-b76c801c0c58/volumes" Nov 01 22:55:25.869260 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:25.869225 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:26.373604 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:26.373571 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht"] Nov 01 22:55:26.373799 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:26.373766 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" podUID="35f7128f-2e3a-4c22-ad5f-9ee3b219c89d" containerName="istio-proxy" containerID="cri-o://010146e0b492f013ded85565cd2e87cc147e1ee0ef7f44fbd29eb721a24a4d48" gracePeriod=30 Nov 01 22:55:26.875416 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:26.875382 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:27.665864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:27.661770 5011 patch_prober.go:29] interesting pod/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:55:27.665864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:27.661855 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" podUID="35f7128f-2e3a-4c22-ad5f-9ee3b219c89d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:55:27.855860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:27.855786 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:27.856072 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:27.855914 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:55:31.692449 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.692416 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:55:31.799967 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.799923 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-credential-socket\") pod \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " Nov 01 22:55:31.800179 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.800116 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istio-token\" (UniqueName: \"kubernetes.io/projected/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istio-token\") pod \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " Nov 01 22:55:31.800241 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.800187 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istio-podinfo\") pod \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " Nov 01 22:55:31.800300 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.800281 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istiod-ca-cert\") pod \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " Nov 01 22:55:31.800356 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.800338 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-workload-certs\") pod \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " Nov 01 22:55:31.800403 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.800379 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istio-data\") pod \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " Nov 01 22:55:31.804328 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.800484 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-workload-socket\") pod \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " Nov 01 22:55:31.804328 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.800540 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istio-envoy\") pod \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " Nov 01 22:55:31.804328 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.800599 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-kube-api-access-62m66\" (UniqueName: \"kubernetes.io/projected/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-kube-api-access-62m66\") pod \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\" (UID: \"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d\") " Nov 01 22:55:31.805497 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.805461 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-credential-socket" pod "35f7128f-2e3a-4c22-ad5f-9ee3b219c89d" (UID: "35f7128f-2e3a-4c22-ad5f-9ee3b219c89d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:31.805774 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.805488 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istiod-ca-cert" pod "35f7128f-2e3a-4c22-ad5f-9ee3b219c89d" (UID: "35f7128f-2e3a-4c22-ad5f-9ee3b219c89d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:55:31.805904 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.805735 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istio-data" pod "35f7128f-2e3a-4c22-ad5f-9ee3b219c89d" (UID: "35f7128f-2e3a-4c22-ad5f-9ee3b219c89d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:31.805980 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.805933 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-workload-certs" pod "35f7128f-2e3a-4c22-ad5f-9ee3b219c89d" (UID: "35f7128f-2e3a-4c22-ad5f-9ee3b219c89d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:31.806118 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.806096 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-workload-socket" pod "35f7128f-2e3a-4c22-ad5f-9ee3b219c89d" (UID: "35f7128f-2e3a-4c22-ad5f-9ee3b219c89d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:31.806254 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.806198 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-kube-api-access-62m66" pod "35f7128f-2e3a-4c22-ad5f-9ee3b219c89d" (UID: "35f7128f-2e3a-4c22-ad5f-9ee3b219c89d"). InnerVolumeSpecName "kube-api-access-62m66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:31.808262 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.808233 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istio-podinfo" pod "35f7128f-2e3a-4c22-ad5f-9ee3b219c89d" (UID: "35f7128f-2e3a-4c22-ad5f-9ee3b219c89d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:55:31.808405 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.808380 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istio-envoy" pod "35f7128f-2e3a-4c22-ad5f-9ee3b219c89d" (UID: "35f7128f-2e3a-4c22-ad5f-9ee3b219c89d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:31.808679 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.808657 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istio-token" pod "35f7128f-2e3a-4c22-ad5f-9ee3b219c89d" (UID: "35f7128f-2e3a-4c22-ad5f-9ee3b219c89d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:31.847211 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.847172 5011 generic.go:356] "Generic (PLEG): container finished" podID="35f7128f-2e3a-4c22-ad5f-9ee3b219c89d" containerID="010146e0b492f013ded85565cd2e87cc147e1ee0ef7f44fbd29eb721a24a4d48" exitCode=0 Nov 01 22:55:31.847413 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.847269 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" Nov 01 22:55:31.847413 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.847268 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" event={"ID":"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d","Type":"ContainerDied","Data":"010146e0b492f013ded85565cd2e87cc147e1ee0ef7f44fbd29eb721a24a4d48"} Nov 01 22:55:31.847413 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.847369 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht" event={"ID":"35f7128f-2e3a-4c22-ad5f-9ee3b219c89d","Type":"ContainerDied","Data":"dcdda9ae4c36e3367a4ffbe6a052b5ece36d1630d91033e250490e2fb491f803"} Nov 01 22:55:31.847543 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.847411 5011 scope.go:122] "RemoveContainer" containerID="010146e0b492f013ded85565cd2e87cc147e1ee0ef7f44fbd29eb721a24a4d48" Nov 01 22:55:31.861178 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.861159 5011 scope.go:122] "RemoveContainer" containerID="010146e0b492f013ded85565cd2e87cc147e1ee0ef7f44fbd29eb721a24a4d48" Nov 01 22:55:31.861510 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:55:31.861482 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"010146e0b492f013ded85565cd2e87cc147e1ee0ef7f44fbd29eb721a24a4d48\": container with ID starting with 010146e0b492f013ded85565cd2e87cc147e1ee0ef7f44fbd29eb721a24a4d48 not found: ID does not exist" containerID="010146e0b492f013ded85565cd2e87cc147e1ee0ef7f44fbd29eb721a24a4d48" Nov 01 22:55:31.861571 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.861527 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"010146e0b492f013ded85565cd2e87cc147e1ee0ef7f44fbd29eb721a24a4d48"} err="failed to get container status \"010146e0b492f013ded85565cd2e87cc147e1ee0ef7f44fbd29eb721a24a4d48\": rpc error: code = NotFound desc = could not find container \"010146e0b492f013ded85565cd2e87cc147e1ee0ef7f44fbd29eb721a24a4d48\": container with ID starting with 010146e0b492f013ded85565cd2e87cc147e1ee0ef7f44fbd29eb721a24a4d48 not found: ID does not exist" Nov 01 22:55:31.883473 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.883443 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht"] Nov 01 22:55:31.896632 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.896557 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-plkht"] Nov 01 22:55:31.903619 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.903583 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:31.903619 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.903620 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:31.903855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.903640 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62m66\" (UniqueName: \"kubernetes.io/projected/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-kube-api-access-62m66\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:31.903855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.903666 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:31.903855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.903681 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:31.903855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.903711 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:31.903855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.903734 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:31.903855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.903747 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:31.903855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:31.903760 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:33.750260 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:33.750227 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35f7128f-2e3a-4c22-ad5f-9ee3b219c89d" path="/var/lib/kubelet/pods/35f7128f-2e3a-4c22-ad5f-9ee3b219c89d/volumes" Nov 01 22:55:34.307169 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:34.307133 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-49ab43-696d67bfd7-mkrrg"] Nov 01 22:55:34.308807 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:34.308781 5011 cpu_manager.go:414] "containerMap: removing container" podUID="35f7128f-2e3a-4c22-ad5f-9ee3b219c89d" containerName="istio-proxy" Nov 01 22:55:34.308991 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:34.308966 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35f7128f-2e3a-4c22-ad5f-9ee3b219c89d" containerName="istio-proxy" Nov 01 22:55:34.309265 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:34.309250 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="35f7128f-2e3a-4c22-ad5f-9ee3b219c89d" containerName="istio-proxy" Nov 01 22:55:34.314792 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:34.314772 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-49ab43-696d67bfd7-mkrrg" Nov 01 22:55:34.329015 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:34.328989 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-49ab43-696d67bfd7-mkrrg"] Nov 01 22:55:34.329780 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:34.329740 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-49ab43\"" Nov 01 22:55:34.343197 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:34.343166 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-49ab43-5bc95ddff9-m8bzq"] Nov 01 22:55:34.349059 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:34.349037 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-49ab43-5bc95ddff9-m8bzq" Nov 01 22:55:34.373121 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:34.373083 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-49ab43-dockercfg-f2js5\"" Nov 01 22:55:34.375486 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:34.375462 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-49ab43-5bc95ddff9-m8bzq"] Nov 01 22:55:34.508852 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:34.504012 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c828f45b-c1cd-4f21-8ab1-b050cba1a54e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-49ab43-5bc95ddff9-m8bzq\" (UID: \"c828f45b-c1cd-4f21-8ab1-b050cba1a54e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-49ab43-5bc95ddff9-m8bzq" Nov 01 22:55:34.508852 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:34.504309 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqrzt\" (UniqueName: \"kubernetes.io/projected/c828f45b-c1cd-4f21-8ab1-b050cba1a54e-kube-api-access-jqrzt\") pod \"payload-pre-processing-e2e-shared-a-w6-49ab43-5bc95ddff9-m8bzq\" (UID: \"c828f45b-c1cd-4f21-8ab1-b050cba1a54e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-49ab43-5bc95ddff9-m8bzq" Nov 01 22:55:34.508852 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:34.504378 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e3321d-ea40-40e4-9c13-d55c83c02e61-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-49ab43-696d67bfd7-mkrrg\" (UID: \"94e3321d-ea40-40e4-9c13-d55c83c02e61\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-49ab43-696d67bfd7-mkrrg" Nov 01 22:55:34.508852 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:34.504435 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmtsw\" (UniqueName: \"kubernetes.io/projected/94e3321d-ea40-40e4-9c13-d55c83c02e61-kube-api-access-pmtsw\") pod \"payload-processing-e2e-shared-a-w6-49ab43-696d67bfd7-mkrrg\" (UID: \"94e3321d-ea40-40e4-9c13-d55c83c02e61\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-49ab43-696d67bfd7-mkrrg" Nov 01 22:55:34.612104 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:34.609371 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqrzt\" (UniqueName: \"kubernetes.io/projected/c828f45b-c1cd-4f21-8ab1-b050cba1a54e-kube-api-access-jqrzt\") pod \"payload-pre-processing-e2e-shared-a-w6-49ab43-5bc95ddff9-m8bzq\" (UID: \"c828f45b-c1cd-4f21-8ab1-b050cba1a54e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-49ab43-5bc95ddff9-m8bzq" Nov 01 22:55:34.612104 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:34.609449 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e3321d-ea40-40e4-9c13-d55c83c02e61-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-49ab43-696d67bfd7-mkrrg\" (UID: \"94e3321d-ea40-40e4-9c13-d55c83c02e61\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-49ab43-696d67bfd7-mkrrg" Nov 01 22:55:34.612104 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:34.609505 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmtsw\" (UniqueName: \"kubernetes.io/projected/94e3321d-ea40-40e4-9c13-d55c83c02e61-kube-api-access-pmtsw\") pod \"payload-processing-e2e-shared-a-w6-49ab43-696d67bfd7-mkrrg\" (UID: \"94e3321d-ea40-40e4-9c13-d55c83c02e61\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-49ab43-696d67bfd7-mkrrg" Nov 01 22:55:34.612104 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:34.609616 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c828f45b-c1cd-4f21-8ab1-b050cba1a54e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-49ab43-5bc95ddff9-m8bzq\" (UID: \"c828f45b-c1cd-4f21-8ab1-b050cba1a54e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-49ab43-5bc95ddff9-m8bzq" Nov 01 22:55:34.612104 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:34.610799 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c828f45b-c1cd-4f21-8ab1-b050cba1a54e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-49ab43-5bc95ddff9-m8bzq\" (UID: \"c828f45b-c1cd-4f21-8ab1-b050cba1a54e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-49ab43-5bc95ddff9-m8bzq" Nov 01 22:55:34.644857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:34.624749 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e3321d-ea40-40e4-9c13-d55c83c02e61-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-49ab43-696d67bfd7-mkrrg\" (UID: \"94e3321d-ea40-40e4-9c13-d55c83c02e61\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-49ab43-696d67bfd7-mkrrg" Nov 01 22:55:34.660641 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:34.647899 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqrzt\" (UniqueName: \"kubernetes.io/projected/c828f45b-c1cd-4f21-8ab1-b050cba1a54e-kube-api-access-jqrzt\") pod \"payload-pre-processing-e2e-shared-a-w6-49ab43-5bc95ddff9-m8bzq\" (UID: \"c828f45b-c1cd-4f21-8ab1-b050cba1a54e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-49ab43-5bc95ddff9-m8bzq" Nov 01 22:55:34.660641 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:34.660549 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmtsw\" (UniqueName: \"kubernetes.io/projected/94e3321d-ea40-40e4-9c13-d55c83c02e61-kube-api-access-pmtsw\") pod \"payload-processing-e2e-shared-a-w6-49ab43-696d67bfd7-mkrrg\" (UID: \"94e3321d-ea40-40e4-9c13-d55c83c02e61\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-49ab43-696d67bfd7-mkrrg" Nov 01 22:55:34.681938 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:34.676312 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-49ab43-5bc95ddff9-m8bzq" Nov 01 22:55:34.935611 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:34.935293 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-49ab43-696d67bfd7-mkrrg" Nov 01 22:55:35.155072 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:35.143330 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-49ab43-5bc95ddff9-m8bzq"] Nov 01 22:55:35.320589 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:55:35.320554 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e3321d_ea40_40e4_9c13_d55c83c02e61.slice/crio-6a38bd44a40f5a773da1cb4ad0f7e323530a19848d070a982ce66cb607e4c5ab WatchSource:0}: Error finding container 6a38bd44a40f5a773da1cb4ad0f7e323530a19848d070a982ce66cb607e4c5ab: Status 404 returned error can't find the container with id 6a38bd44a40f5a773da1cb4ad0f7e323530a19848d070a982ce66cb607e4c5ab Nov 01 22:55:35.328126 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:35.328097 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-49ab43-696d67bfd7-mkrrg"] Nov 01 22:55:35.912306 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:35.912267 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-49ab43-696d67bfd7-mkrrg" event={"ID":"94e3321d-ea40-40e4-9c13-d55c83c02e61","Type":"ContainerStarted","Data":"1e851d4e1015415514f5223640bb8bc9be71f80b21450401c4f147409cf92d10"} Nov 01 22:55:35.912498 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:35.912328 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-49ab43-696d67bfd7-mkrrg" event={"ID":"94e3321d-ea40-40e4-9c13-d55c83c02e61","Type":"ContainerStarted","Data":"6a38bd44a40f5a773da1cb4ad0f7e323530a19848d070a982ce66cb607e4c5ab"} Nov 01 22:55:35.912498 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:35.912359 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-49ab43-696d67bfd7-mkrrg" Nov 01 22:55:35.914268 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:35.914241 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-49ab43-5bc95ddff9-m8bzq" event={"ID":"c828f45b-c1cd-4f21-8ab1-b050cba1a54e","Type":"ContainerStarted","Data":"a72ec049be112cb0903f4f3108f301af12db7222bf7461520215faafd447c212"} Nov 01 22:55:35.914421 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:35.914292 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-49ab43-5bc95ddff9-m8bzq" event={"ID":"c828f45b-c1cd-4f21-8ab1-b050cba1a54e","Type":"ContainerStarted","Data":"f072eae0faeb0a9f8f01a9bc3ae64d43eac097ce2c117618f6ef1d0cb296694f"} Nov 01 22:55:35.914421 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:35.914322 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-49ab43-5bc95ddff9-m8bzq" Nov 01 22:55:35.929084 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:35.929034 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-49ab43-696d67bfd7-mkrrg" podStartSLOduration=1.929021021 podStartE2EDuration="1.929021021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:35.927982964 +0000 UTC m=+1904.332573620" watchObservedRunningTime="2026-11-01 22:55:35.929021021 +0000 UTC m=+1904.333611653" Nov 01 22:55:35.947214 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:35.947165 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-49ab43-5bc95ddff9-m8bzq" podStartSLOduration=1.947151795 podStartE2EDuration="1.947151795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:35.943586882 +0000 UTC m=+1904.348177515" watchObservedRunningTime="2026-11-01 22:55:35.947151795 +0000 UTC m=+1904.351742426" Nov 01 22:55:46.953262 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:46.953228 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-49ab43-696d67bfd7-mkrrg" Nov 01 22:55:46.954789 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:46.954709 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-49ab43-5bc95ddff9-m8bzq" Nov 01 22:55:46.980480 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:55:46.980451 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:55:47.775216 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:47.775175 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-58c957-6bf77f96c5-lgnfs"] Nov 01 22:55:52.776434 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:52.776393 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-58c957-6bf77f96c5-lgnfs" podUID="a09edcbb-afbe-471a-8f37-001071ed1aec" containerName="payload-pre-processing" containerID="cri-o://8d8a9af1fbb335cb6de7fbee31787f1ff030501eda1f2e941a8497b15980188f" gracePeriod=25 Nov 01 22:55:54.408856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:54.407425 5011 scope.go:122] "RemoveContainer" containerID="40f1dfe0d3bcc9c2b39c3bb22ab045fe55bfee32124afd322207f6d04f652bd2" Nov 01 22:55:54.483304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:54.483268 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-58c957-6bf77f96c5-lgnfs" podUID="a09edcbb-afbe-471a-8f37-001071ed1aec" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:55:54.540534 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:54.540505 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e202e7-595b687479-zhkww"] Nov 01 22:55:54.553869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:54.553821 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e202e7-58d7c8ff6c-6gzzg"] Nov 01 22:55:56.317989 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:56.317948 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8"] Nov 01 22:55:56.351544 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:56.351514 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8"] Nov 01 22:55:56.351875 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:56.351858 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8" Nov 01 22:55:56.358795 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:56.358763 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-49ab43-serving-cert\"" Nov 01 22:55:56.502352 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:56.496064 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56973f4-5487-4aa0-88e9-c0e0161fb912-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8\" (UID: \"c56973f4-5487-4aa0-88e9-c0e0161fb912\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8" Nov 01 22:55:56.522994 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:56.512981 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56973f4-5487-4aa0-88e9-c0e0161fb912-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8\" (UID: \"c56973f4-5487-4aa0-88e9-c0e0161fb912\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8" Nov 01 22:55:56.522994 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:56.513167 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbmtx\" (UniqueName: \"kubernetes.io/projected/c56973f4-5487-4aa0-88e9-c0e0161fb912-kube-api-access-tbmtx\") pod \"maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8\" (UID: \"c56973f4-5487-4aa0-88e9-c0e0161fb912\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8" Nov 01 22:55:56.522994 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:56.513289 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c56973f4-5487-4aa0-88e9-c0e0161fb912-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8\" (UID: \"c56973f4-5487-4aa0-88e9-c0e0161fb912\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8" Nov 01 22:55:56.614754 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:56.614667 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56973f4-5487-4aa0-88e9-c0e0161fb912-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8\" (UID: \"c56973f4-5487-4aa0-88e9-c0e0161fb912\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8" Nov 01 22:55:56.614948 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:56.614780 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56973f4-5487-4aa0-88e9-c0e0161fb912-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8\" (UID: \"c56973f4-5487-4aa0-88e9-c0e0161fb912\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8" Nov 01 22:55:56.614948 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:56.614880 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbmtx\" (UniqueName: \"kubernetes.io/projected/c56973f4-5487-4aa0-88e9-c0e0161fb912-kube-api-access-tbmtx\") pod \"maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8\" (UID: \"c56973f4-5487-4aa0-88e9-c0e0161fb912\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8" Nov 01 22:55:56.614948 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:56.614943 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c56973f4-5487-4aa0-88e9-c0e0161fb912-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8\" (UID: \"c56973f4-5487-4aa0-88e9-c0e0161fb912\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8" Nov 01 22:55:56.615499 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:56.615472 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56973f4-5487-4aa0-88e9-c0e0161fb912-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8\" (UID: \"c56973f4-5487-4aa0-88e9-c0e0161fb912\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8" Nov 01 22:55:56.617439 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:56.617416 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56973f4-5487-4aa0-88e9-c0e0161fb912-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8\" (UID: \"c56973f4-5487-4aa0-88e9-c0e0161fb912\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8" Nov 01 22:55:56.617439 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:56.617435 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c56973f4-5487-4aa0-88e9-c0e0161fb912-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8\" (UID: \"c56973f4-5487-4aa0-88e9-c0e0161fb912\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8" Nov 01 22:55:56.624022 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:56.624001 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbmtx\" (UniqueName: \"kubernetes.io/projected/c56973f4-5487-4aa0-88e9-c0e0161fb912-kube-api-access-tbmtx\") pod \"maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8\" (UID: \"c56973f4-5487-4aa0-88e9-c0e0161fb912\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8" Nov 01 22:55:56.672373 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:56.672339 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8" Nov 01 22:55:57.092824 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:57.091915 5011 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:55:57.124885 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:57.122351 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8"] Nov 01 22:55:57.275714 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:57.275668 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8" event={"ID":"c56973f4-5487-4aa0-88e9-c0e0161fb912","Type":"ContainerStarted","Data":"94dfd39a0c678fa7e12785cbefa07924b2adb534e34a798af84d6d91f5fffc35"} Nov 01 22:55:58.285578 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:58.285545 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8" event={"ID":"c56973f4-5487-4aa0-88e9-c0e0161fb912","Type":"ContainerStarted","Data":"3412f6d4012884f829c0c8d6bfe2256c6ec4a743943d3ed56fa1edb57105bc55"} Nov 01 22:55:58.286062 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:58.285644 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8" Nov 01 22:55:58.305032 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:58.304988 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8" podStartSLOduration=1.78388495 podStartE2EDuration="2.304974394s" totalImagesPullingTime="521.089444ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:55:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:58.303342757 +0000 UTC m=+1926.707933389" watchObservedRunningTime="2026-11-01 22:55:58.304974394 +0000 UTC m=+1926.709565025" Nov 01 22:55:59.541169 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:59.541107 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-e202e7-595b687479-zhkww" podUID="583600ad-f7bb-49e2-898c-2bd002a6b1d6" containerName="payload-processing" containerID="cri-o://51049bc1781caaff785e307ac425ad2c6a60d8002243726b388938e15978393b" gracePeriod=25 Nov 01 22:55:59.554680 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:59.554645 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e202e7-58d7c8ff6c-6gzzg" podUID="1c8ba82e-675b-48ee-bc33-d683b2c171cc" containerName="payload-pre-processing" containerID="cri-o://7fecf81348812edb1f021717a7622a386b9297d316895a4793d14f79f4219ab0" gracePeriod=25 Nov 01 22:55:59.866162 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:59.866133 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e202e7-595b687479-zhkww" Nov 01 22:55:59.966887 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:59.966846 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/583600ad-f7bb-49e2-898c-2bd002a6b1d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/583600ad-f7bb-49e2-898c-2bd002a6b1d6-plugins-config-volume\") pod \"583600ad-f7bb-49e2-898c-2bd002a6b1d6\" (UID: \"583600ad-f7bb-49e2-898c-2bd002a6b1d6\") " Nov 01 22:55:59.967111 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:59.967092 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/583600ad-f7bb-49e2-898c-2bd002a6b1d6-kube-api-access-9vjhw\" (UniqueName: \"kubernetes.io/projected/583600ad-f7bb-49e2-898c-2bd002a6b1d6-kube-api-access-9vjhw\") pod \"583600ad-f7bb-49e2-898c-2bd002a6b1d6\" (UID: \"583600ad-f7bb-49e2-898c-2bd002a6b1d6\") " Nov 01 22:55:59.967190 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:59.967140 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/583600ad-f7bb-49e2-898c-2bd002a6b1d6-plugins-config-volume" pod "583600ad-f7bb-49e2-898c-2bd002a6b1d6" (UID: "583600ad-f7bb-49e2-898c-2bd002a6b1d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:55:59.967818 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:59.967795 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/583600ad-f7bb-49e2-898c-2bd002a6b1d6-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:59.969390 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:55:59.969369 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583600ad-f7bb-49e2-898c-2bd002a6b1d6-kube-api-access-9vjhw" pod "583600ad-f7bb-49e2-898c-2bd002a6b1d6" (UID: "583600ad-f7bb-49e2-898c-2bd002a6b1d6"). InnerVolumeSpecName "kube-api-access-9vjhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:00.069317 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:00.069285 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vjhw\" (UniqueName: \"kubernetes.io/projected/583600ad-f7bb-49e2-898c-2bd002a6b1d6-kube-api-access-9vjhw\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:00.306601 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:00.306557 5011 generic.go:356] "Generic (PLEG): container finished" podID="583600ad-f7bb-49e2-898c-2bd002a6b1d6" containerID="51049bc1781caaff785e307ac425ad2c6a60d8002243726b388938e15978393b" exitCode=0 Nov 01 22:56:00.306787 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:00.306619 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e202e7-595b687479-zhkww" Nov 01 22:56:00.306787 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:00.306629 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e202e7-595b687479-zhkww" event={"ID":"583600ad-f7bb-49e2-898c-2bd002a6b1d6","Type":"ContainerDied","Data":"51049bc1781caaff785e307ac425ad2c6a60d8002243726b388938e15978393b"} Nov 01 22:56:00.306787 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:00.306683 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e202e7-595b687479-zhkww" event={"ID":"583600ad-f7bb-49e2-898c-2bd002a6b1d6","Type":"ContainerDied","Data":"771e46ad54e5189a338ffa7bfdc39675d7417b2ca1afb75d5db937afbd8d3bb4"} Nov 01 22:56:00.306787 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:00.306712 5011 scope.go:122] "RemoveContainer" containerID="51049bc1781caaff785e307ac425ad2c6a60d8002243726b388938e15978393b" Nov 01 22:56:00.320997 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:00.320302 5011 scope.go:122] "RemoveContainer" containerID="51049bc1781caaff785e307ac425ad2c6a60d8002243726b388938e15978393b" Nov 01 22:56:00.323032 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:56:00.323007 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51049bc1781caaff785e307ac425ad2c6a60d8002243726b388938e15978393b\": container with ID starting with 51049bc1781caaff785e307ac425ad2c6a60d8002243726b388938e15978393b not found: ID does not exist" containerID="51049bc1781caaff785e307ac425ad2c6a60d8002243726b388938e15978393b" Nov 01 22:56:00.323149 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:00.323039 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51049bc1781caaff785e307ac425ad2c6a60d8002243726b388938e15978393b"} err="failed to get container status \"51049bc1781caaff785e307ac425ad2c6a60d8002243726b388938e15978393b\": rpc error: code = NotFound desc = could not find container \"51049bc1781caaff785e307ac425ad2c6a60d8002243726b388938e15978393b\": container with ID starting with 51049bc1781caaff785e307ac425ad2c6a60d8002243726b388938e15978393b not found: ID does not exist" Nov 01 22:56:00.333017 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:00.332993 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e202e7-595b687479-zhkww"] Nov 01 22:56:00.338604 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:00.338576 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e202e7-595b687479-zhkww"] Nov 01 22:56:01.752602 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:01.752565 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="583600ad-f7bb-49e2-898c-2bd002a6b1d6" path="/var/lib/kubelet/pods/583600ad-f7bb-49e2-898c-2bd002a6b1d6/volumes" Nov 01 22:56:04.325863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:04.319849 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8" Nov 01 22:56:04.483159 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:04.483120 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-58c957-6bf77f96c5-lgnfs" podUID="a09edcbb-afbe-471a-8f37-001071ed1aec" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:56:06.635146 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:06.635100 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e202e7-58d7c8ff6c-6gzzg" podUID="1c8ba82e-675b-48ee-bc33-d683b2c171cc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:56:12.878002 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:12.876974 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:12.878002 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:12.877679 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 01 22:56:13.673454 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:13.667908 5011 generic.go:356] "Generic (PLEG): container finished" podID="a09edcbb-afbe-471a-8f37-001071ed1aec" containerID="8d8a9af1fbb335cb6de7fbee31787f1ff030501eda1f2e941a8497b15980188f" exitCode=0 Nov 01 22:56:13.673454 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:13.668118 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-58c957-6bf77f96c5-lgnfs" event={"ID":"a09edcbb-afbe-471a-8f37-001071ed1aec","Type":"ContainerDied","Data":"8d8a9af1fbb335cb6de7fbee31787f1ff030501eda1f2e941a8497b15980188f"} Nov 01 22:56:14.318548 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:14.318518 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-58c957-6bf77f96c5-lgnfs" Nov 01 22:56:14.407636 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:14.407546 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a09edcbb-afbe-471a-8f37-001071ed1aec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a09edcbb-afbe-471a-8f37-001071ed1aec-plugins-config-volume\") pod \"a09edcbb-afbe-471a-8f37-001071ed1aec\" (UID: \"a09edcbb-afbe-471a-8f37-001071ed1aec\") " Nov 01 22:56:14.407953 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:14.407925 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a09edcbb-afbe-471a-8f37-001071ed1aec-kube-api-access-86zsv\" (UniqueName: \"kubernetes.io/projected/a09edcbb-afbe-471a-8f37-001071ed1aec-kube-api-access-86zsv\") pod \"a09edcbb-afbe-471a-8f37-001071ed1aec\" (UID: \"a09edcbb-afbe-471a-8f37-001071ed1aec\") " Nov 01 22:56:14.409640 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:14.409607 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a09edcbb-afbe-471a-8f37-001071ed1aec-plugins-config-volume" pod "a09edcbb-afbe-471a-8f37-001071ed1aec" (UID: "a09edcbb-afbe-471a-8f37-001071ed1aec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:56:14.412593 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:14.412564 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a09edcbb-afbe-471a-8f37-001071ed1aec-kube-api-access-86zsv" pod "a09edcbb-afbe-471a-8f37-001071ed1aec" (UID: "a09edcbb-afbe-471a-8f37-001071ed1aec"). InnerVolumeSpecName "kube-api-access-86zsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:14.510189 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:14.510150 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a09edcbb-afbe-471a-8f37-001071ed1aec-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:14.510189 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:14.510183 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86zsv\" (UniqueName: \"kubernetes.io/projected/a09edcbb-afbe-471a-8f37-001071ed1aec-kube-api-access-86zsv\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:14.728898 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:14.728870 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-58c957-6bf77f96c5-lgnfs" Nov 01 22:56:14.731981 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:14.729013 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-58c957-6bf77f96c5-lgnfs" event={"ID":"a09edcbb-afbe-471a-8f37-001071ed1aec","Type":"ContainerDied","Data":"76273ec68fccf079bc2f0e3c996a1fc4fd5fb8b863a3cd1ace08dca4abd1d017"} Nov 01 22:56:14.732543 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:14.732026 5011 scope.go:122] "RemoveContainer" containerID="8d8a9af1fbb335cb6de7fbee31787f1ff030501eda1f2e941a8497b15980188f" Nov 01 22:56:14.793929 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:14.793799 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-58c957-6bf77f96c5-lgnfs"] Nov 01 22:56:14.801725 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:14.801691 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-58c957-6bf77f96c5-lgnfs"] Nov 01 22:56:15.291913 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:15.289068 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e202e7-687db7885-b9p7c"] Nov 01 22:56:15.291913 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:15.289320 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e202e7-687db7885-b9p7c" podUID="41ea1680-d0bb-4926-bb24-3322cf8eeda8" containerName="maas-api" containerID="cri-o://dba42f2a24dde556181d17b6f4b21a768859841a1d0d3ab53a88f800b311450a" gracePeriod=30 Nov 01 22:56:16.044645 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:16.043808 5011 generic.go:356] "Generic (PLEG): container finished" podID="41ea1680-d0bb-4926-bb24-3322cf8eeda8" containerID="dba42f2a24dde556181d17b6f4b21a768859841a1d0d3ab53a88f800b311450a" exitCode=0 Nov 01 22:56:16.129787 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:16.129755 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e202e7-687db7885-b9p7c" Nov 01 22:56:16.154857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:16.140941 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a09edcbb-afbe-471a-8f37-001071ed1aec" path="/var/lib/kubelet/pods/a09edcbb-afbe-471a-8f37-001071ed1aec/volumes" Nov 01 22:56:16.154857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:16.141442 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e202e7-687db7885-b9p7c" event={"ID":"41ea1680-d0bb-4926-bb24-3322cf8eeda8","Type":"ContainerDied","Data":"dba42f2a24dde556181d17b6f4b21a768859841a1d0d3ab53a88f800b311450a"} Nov 01 22:56:16.154857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:16.141508 5011 scope.go:122] "RemoveContainer" containerID="dba42f2a24dde556181d17b6f4b21a768859841a1d0d3ab53a88f800b311450a" Nov 01 22:56:16.275580 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:16.275433 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41ea1680-d0bb-4926-bb24-3322cf8eeda8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/41ea1680-d0bb-4926-bb24-3322cf8eeda8-metrics-certs\") pod \"41ea1680-d0bb-4926-bb24-3322cf8eeda8\" (UID: \"41ea1680-d0bb-4926-bb24-3322cf8eeda8\") " Nov 01 22:56:16.275580 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:16.275551 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41ea1680-d0bb-4926-bb24-3322cf8eeda8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41ea1680-d0bb-4926-bb24-3322cf8eeda8-openshift-service-ca\") pod \"41ea1680-d0bb-4926-bb24-3322cf8eeda8\" (UID: \"41ea1680-d0bb-4926-bb24-3322cf8eeda8\") " Nov 01 22:56:16.275867 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:16.275632 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41ea1680-d0bb-4926-bb24-3322cf8eeda8-kube-api-access-z8scp\" (UniqueName: \"kubernetes.io/projected/41ea1680-d0bb-4926-bb24-3322cf8eeda8-kube-api-access-z8scp\") pod \"41ea1680-d0bb-4926-bb24-3322cf8eeda8\" (UID: \"41ea1680-d0bb-4926-bb24-3322cf8eeda8\") " Nov 01 22:56:16.275867 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:16.275652 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41ea1680-d0bb-4926-bb24-3322cf8eeda8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41ea1680-d0bb-4926-bb24-3322cf8eeda8-maas-api-tls\") pod \"41ea1680-d0bb-4926-bb24-3322cf8eeda8\" (UID: \"41ea1680-d0bb-4926-bb24-3322cf8eeda8\") " Nov 01 22:56:16.276734 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:16.276699 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41ea1680-d0bb-4926-bb24-3322cf8eeda8-openshift-service-ca" pod "41ea1680-d0bb-4926-bb24-3322cf8eeda8" (UID: "41ea1680-d0bb-4926-bb24-3322cf8eeda8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:56:16.279667 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:16.279640 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41ea1680-d0bb-4926-bb24-3322cf8eeda8-metrics-certs" pod "41ea1680-d0bb-4926-bb24-3322cf8eeda8" (UID: "41ea1680-d0bb-4926-bb24-3322cf8eeda8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:56:16.279793 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:16.279694 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ea1680-d0bb-4926-bb24-3322cf8eeda8-kube-api-access-z8scp" pod "41ea1680-d0bb-4926-bb24-3322cf8eeda8" (UID: "41ea1680-d0bb-4926-bb24-3322cf8eeda8"). InnerVolumeSpecName "kube-api-access-z8scp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:16.284146 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:16.284124 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41ea1680-d0bb-4926-bb24-3322cf8eeda8-maas-api-tls" pod "41ea1680-d0bb-4926-bb24-3322cf8eeda8" (UID: "41ea1680-d0bb-4926-bb24-3322cf8eeda8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:56:16.384877 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:16.384310 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41ea1680-d0bb-4926-bb24-3322cf8eeda8-metrics-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:16.384877 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:16.384357 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41ea1680-d0bb-4926-bb24-3322cf8eeda8-openshift-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:16.384877 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:16.384376 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8scp\" (UniqueName: \"kubernetes.io/projected/41ea1680-d0bb-4926-bb24-3322cf8eeda8-kube-api-access-z8scp\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:16.384877 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:16.384390 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41ea1680-d0bb-4926-bb24-3322cf8eeda8-maas-api-tls\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:16.636071 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:16.635990 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e202e7-58d7c8ff6c-6gzzg" podUID="1c8ba82e-675b-48ee-bc33-d683b2c171cc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:56:17.092963 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:17.092919 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e202e7-687db7885-b9p7c" event={"ID":"41ea1680-d0bb-4926-bb24-3322cf8eeda8","Type":"ContainerDied","Data":"73028741a178e527bb90bff6a9b233045c573b6c0205b30e6fb4e3efdb102848"} Nov 01 22:56:17.094480 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:17.094423 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e202e7-687db7885-b9p7c" Nov 01 22:56:17.100739 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:56:17.100710 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:56:17.139284 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:17.139242 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e202e7-687db7885-b9p7c"] Nov 01 22:56:17.180004 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:17.179971 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e202e7-687db7885-b9p7c"] Nov 01 22:56:17.764704 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:17.764666 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41ea1680-d0bb-4926-bb24-3322cf8eeda8" path="/var/lib/kubelet/pods/41ea1680-d0bb-4926-bb24-3322cf8eeda8/volumes" Nov 01 22:56:20.001346 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:20.001319 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e202e7-58d7c8ff6c-6gzzg" Nov 01 22:56:20.025787 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:20.025753 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c8ba82e-675b-48ee-bc33-d683b2c171cc-kube-api-access-wlkgl\" (UniqueName: \"kubernetes.io/projected/1c8ba82e-675b-48ee-bc33-d683b2c171cc-kube-api-access-wlkgl\") pod \"1c8ba82e-675b-48ee-bc33-d683b2c171cc\" (UID: \"1c8ba82e-675b-48ee-bc33-d683b2c171cc\") " Nov 01 22:56:20.026005 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:20.025884 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c8ba82e-675b-48ee-bc33-d683b2c171cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c8ba82e-675b-48ee-bc33-d683b2c171cc-plugins-config-volume\") pod \"1c8ba82e-675b-48ee-bc33-d683b2c171cc\" (UID: \"1c8ba82e-675b-48ee-bc33-d683b2c171cc\") " Nov 01 22:56:20.026269 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:20.026241 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c8ba82e-675b-48ee-bc33-d683b2c171cc-plugins-config-volume" pod "1c8ba82e-675b-48ee-bc33-d683b2c171cc" (UID: "1c8ba82e-675b-48ee-bc33-d683b2c171cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:56:20.026643 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:20.026617 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c8ba82e-675b-48ee-bc33-d683b2c171cc-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:20.028514 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:20.028490 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c8ba82e-675b-48ee-bc33-d683b2c171cc-kube-api-access-wlkgl" pod "1c8ba82e-675b-48ee-bc33-d683b2c171cc" (UID: "1c8ba82e-675b-48ee-bc33-d683b2c171cc"). InnerVolumeSpecName "kube-api-access-wlkgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:20.128615 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:20.128578 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlkgl\" (UniqueName: \"kubernetes.io/projected/1c8ba82e-675b-48ee-bc33-d683b2c171cc-kube-api-access-wlkgl\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:20.158325 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:20.158291 5011 generic.go:356] "Generic (PLEG): container finished" podID="1c8ba82e-675b-48ee-bc33-d683b2c171cc" containerID="7fecf81348812edb1f021717a7622a386b9297d316895a4793d14f79f4219ab0" exitCode=0 Nov 01 22:56:20.158554 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:20.158346 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e202e7-58d7c8ff6c-6gzzg" Nov 01 22:56:20.158554 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:20.158373 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e202e7-58d7c8ff6c-6gzzg" event={"ID":"1c8ba82e-675b-48ee-bc33-d683b2c171cc","Type":"ContainerDied","Data":"7fecf81348812edb1f021717a7622a386b9297d316895a4793d14f79f4219ab0"} Nov 01 22:56:20.158554 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:20.158445 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e202e7-58d7c8ff6c-6gzzg" event={"ID":"1c8ba82e-675b-48ee-bc33-d683b2c171cc","Type":"ContainerDied","Data":"5306e156bc29a2ea7ab6e04df14b76a3c802580539126397d48b9f9b0edb6459"} Nov 01 22:56:20.158554 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:20.158477 5011 scope.go:122] "RemoveContainer" containerID="7fecf81348812edb1f021717a7622a386b9297d316895a4793d14f79f4219ab0" Nov 01 22:56:20.172665 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:20.172638 5011 scope.go:122] "RemoveContainer" containerID="7fecf81348812edb1f021717a7622a386b9297d316895a4793d14f79f4219ab0" Nov 01 22:56:20.173091 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:56:20.173063 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fecf81348812edb1f021717a7622a386b9297d316895a4793d14f79f4219ab0\": container with ID starting with 7fecf81348812edb1f021717a7622a386b9297d316895a4793d14f79f4219ab0 not found: ID does not exist" containerID="7fecf81348812edb1f021717a7622a386b9297d316895a4793d14f79f4219ab0" Nov 01 22:56:20.173207 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:20.173096 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fecf81348812edb1f021717a7622a386b9297d316895a4793d14f79f4219ab0"} err="failed to get container status \"7fecf81348812edb1f021717a7622a386b9297d316895a4793d14f79f4219ab0\": rpc error: code = NotFound desc = could not find container \"7fecf81348812edb1f021717a7622a386b9297d316895a4793d14f79f4219ab0\": container with ID starting with 7fecf81348812edb1f021717a7622a386b9297d316895a4793d14f79f4219ab0 not found: ID does not exist" Nov 01 22:56:20.187469 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:20.187442 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e202e7-58d7c8ff6c-6gzzg"] Nov 01 22:56:20.192720 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:20.192692 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e202e7-58d7c8ff6c-6gzzg"] Nov 01 22:56:21.756149 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:21.755729 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c8ba82e-675b-48ee-bc33-d683b2c171cc" path="/var/lib/kubelet/pods/1c8ba82e-675b-48ee-bc33-d683b2c171cc/volumes" Nov 01 22:56:23.252208 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:23.252168 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh"] Nov 01 22:56:23.253005 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:23.252854 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" podUID="34d081d4-e311-40f4-9695-abcc8ff40605" containerName="istio-proxy" containerID="cri-o://ed3342ce267adabf14d5b803ff51e867d579d96a81c437122b619fdbc86c7428" gracePeriod=30 Nov 01 22:56:27.813597 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:27.813550 5011 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 01 22:56:27.814207 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:27.813635 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" podUID="665520e7-d3f3-4dd9-a7a5-3992fd827544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 01 22:56:28.099707 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:28.099621 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0e716d-cbd5484dc-t8bxj"] Nov 01 22:56:28.500858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:28.484418 5011 generic.go:356] "Generic (PLEG): container finished" podID="665520e7-d3f3-4dd9-a7a5-3992fd827544" containerID="dd45e515bc228815d2ab2ea472d08728c59cdb6b2bb7dd947fb3e9ac0a2414bd" exitCode=0 Nov 01 22:56:28.500858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:28.484730 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" event={"ID":"665520e7-d3f3-4dd9-a7a5-3992fd827544","Type":"ContainerDied","Data":"dd45e515bc228815d2ab2ea472d08728c59cdb6b2bb7dd947fb3e9ac0a2414bd"} Nov 01 22:56:28.500858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:28.485289 5011 scope.go:122] "RemoveContainer" containerID="dd45e515bc228815d2ab2ea472d08728c59cdb6b2bb7dd947fb3e9ac0a2414bd" Nov 01 22:56:28.500858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:28.497375 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:56:28.500858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:28.497485 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:56:28.500858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:28.497537 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:56:29.034600 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.034549 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:56:29.153182 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.153103 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-53dd05-5b8b7fd495-b6jn6"] Nov 01 22:56:29.154537 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.154428 5011 cpu_manager.go:414] "containerMap: removing container" podUID="583600ad-f7bb-49e2-898c-2bd002a6b1d6" containerName="payload-processing" Nov 01 22:56:29.154537 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.154452 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="583600ad-f7bb-49e2-898c-2bd002a6b1d6" containerName="payload-processing" Nov 01 22:56:29.154537 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.154478 5011 cpu_manager.go:414] "containerMap: removing container" podUID="a09edcbb-afbe-471a-8f37-001071ed1aec" containerName="payload-pre-processing" Nov 01 22:56:29.154537 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.154486 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a09edcbb-afbe-471a-8f37-001071ed1aec" containerName="payload-pre-processing" Nov 01 22:56:29.154537 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.154504 5011 cpu_manager.go:414] "containerMap: removing container" podUID="34d081d4-e311-40f4-9695-abcc8ff40605" containerName="istio-proxy" Nov 01 22:56:29.154537 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.154511 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34d081d4-e311-40f4-9695-abcc8ff40605" containerName="istio-proxy" Nov 01 22:56:29.154537 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.154533 5011 cpu_manager.go:414] "containerMap: removing container" podUID="1c8ba82e-675b-48ee-bc33-d683b2c171cc" containerName="payload-pre-processing" Nov 01 22:56:29.154537 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.154541 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c8ba82e-675b-48ee-bc33-d683b2c171cc" containerName="payload-pre-processing" Nov 01 22:56:29.155546 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.154626 5011 cpu_manager.go:414] "containerMap: removing container" podUID="41ea1680-d0bb-4926-bb24-3322cf8eeda8" containerName="maas-api" Nov 01 22:56:29.155546 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.154634 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41ea1680-d0bb-4926-bb24-3322cf8eeda8" containerName="maas-api" Nov 01 22:56:29.155546 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.154974 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="34d081d4-e311-40f4-9695-abcc8ff40605" containerName="istio-proxy" Nov 01 22:56:29.155546 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.155018 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c8ba82e-675b-48ee-bc33-d683b2c171cc" containerName="payload-pre-processing" Nov 01 22:56:29.155546 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.155055 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="41ea1680-d0bb-4926-bb24-3322cf8eeda8" containerName="maas-api" Nov 01 22:56:29.155546 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.155067 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="583600ad-f7bb-49e2-898c-2bd002a6b1d6" containerName="payload-processing" Nov 01 22:56:29.155546 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.155080 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="a09edcbb-afbe-471a-8f37-001071ed1aec" containerName="payload-pre-processing" Nov 01 22:56:29.158786 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.158768 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-53dd05-5b8b7fd495-b6jn6" Nov 01 22:56:29.169581 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.167027 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-53dd05-dockercfg-r29wq\"" Nov 01 22:56:29.169581 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.167124 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-53dd05\"" Nov 01 22:56:29.176576 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.176545 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-53dd05-5b8b7fd495-b6jn6"] Nov 01 22:56:29.208533 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.208487 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-53dd05-fd7f899cc-tctb8"] Nov 01 22:56:29.289869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.217498 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34d081d4-e311-40f4-9695-abcc8ff40605-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34d081d4-e311-40f4-9695-abcc8ff40605-istio-podinfo\") pod \"34d081d4-e311-40f4-9695-abcc8ff40605\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " Nov 01 22:56:29.289869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.217561 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-istio-data\") pod \"34d081d4-e311-40f4-9695-abcc8ff40605\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " Nov 01 22:56:29.289869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.217647 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-workload-socket\") pod \"34d081d4-e311-40f4-9695-abcc8ff40605\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " Nov 01 22:56:29.289869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.217699 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-credential-socket\") pod \"34d081d4-e311-40f4-9695-abcc8ff40605\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " Nov 01 22:56:29.289869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.217783 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-workload-certs\") pod \"34d081d4-e311-40f4-9695-abcc8ff40605\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " Nov 01 22:56:29.289869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.217885 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-istio-envoy\") pod \"34d081d4-e311-40f4-9695-abcc8ff40605\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " Nov 01 22:56:29.289869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.217937 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34d081d4-e311-40f4-9695-abcc8ff40605-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34d081d4-e311-40f4-9695-abcc8ff40605-istiod-ca-cert\") pod \"34d081d4-e311-40f4-9695-abcc8ff40605\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " Nov 01 22:56:29.289869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.217969 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34d081d4-e311-40f4-9695-abcc8ff40605-kube-api-access-28mfh\" (UniqueName: \"kubernetes.io/projected/34d081d4-e311-40f4-9695-abcc8ff40605-kube-api-access-28mfh\") pod \"34d081d4-e311-40f4-9695-abcc8ff40605\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " Nov 01 22:56:29.289869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.218338 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34d081d4-e311-40f4-9695-abcc8ff40605-istio-token\" (UniqueName: \"kubernetes.io/projected/34d081d4-e311-40f4-9695-abcc8ff40605-istio-token\") pod \"34d081d4-e311-40f4-9695-abcc8ff40605\" (UID: \"34d081d4-e311-40f4-9695-abcc8ff40605\") " Nov 01 22:56:29.289869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.221925 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-credential-socket" pod "34d081d4-e311-40f4-9695-abcc8ff40605" (UID: "34d081d4-e311-40f4-9695-abcc8ff40605"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:29.289869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.242951 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34d081d4-e311-40f4-9695-abcc8ff40605-istiod-ca-cert" pod "34d081d4-e311-40f4-9695-abcc8ff40605" (UID: "34d081d4-e311-40f4-9695-abcc8ff40605"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:56:29.289869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.256026 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-istio-data" pod "34d081d4-e311-40f4-9695-abcc8ff40605" (UID: "34d081d4-e311-40f4-9695-abcc8ff40605"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:29.289869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.256248 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-workload-socket" pod "34d081d4-e311-40f4-9695-abcc8ff40605" (UID: "34d081d4-e311-40f4-9695-abcc8ff40605"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:29.289869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.256448 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-workload-certs" pod "34d081d4-e311-40f4-9695-abcc8ff40605" (UID: "34d081d4-e311-40f4-9695-abcc8ff40605"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:29.289869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.260344 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-53dd05-fd7f899cc-tctb8" Nov 01 22:56:29.289869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.280492 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34d081d4-e311-40f4-9695-abcc8ff40605-istio-token" pod "34d081d4-e311-40f4-9695-abcc8ff40605" (UID: "34d081d4-e311-40f4-9695-abcc8ff40605"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:29.410249 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.333004 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d5nj\" (UniqueName: \"kubernetes.io/projected/5dbf96e5-22cf-4e5b-8208-470f9c1f616c-kube-api-access-4d5nj\") pod \"payload-pre-processing-e2e-shared-b-w6-53dd05-5b8b7fd495-b6jn6\" (UID: \"5dbf96e5-22cf-4e5b-8208-470f9c1f616c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-53dd05-5b8b7fd495-b6jn6" Nov 01 22:56:29.410249 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.333145 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dbf96e5-22cf-4e5b-8208-470f9c1f616c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-53dd05-5b8b7fd495-b6jn6\" (UID: \"5dbf96e5-22cf-4e5b-8208-470f9c1f616c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-53dd05-5b8b7fd495-b6jn6" Nov 01 22:56:29.410249 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.333457 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34d081d4-e311-40f4-9695-abcc8ff40605-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:29.410249 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.333480 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:29.410249 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.333496 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:29.410249 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.333512 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:29.410249 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.333528 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:29.410249 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.333546 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34d081d4-e311-40f4-9695-abcc8ff40605-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:29.410249 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.373945 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34d081d4-e311-40f4-9695-abcc8ff40605-istio-podinfo" pod "34d081d4-e311-40f4-9695-abcc8ff40605" (UID: "34d081d4-e311-40f4-9695-abcc8ff40605"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:56:29.410249 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.384431 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34d081d4-e311-40f4-9695-abcc8ff40605-kube-api-access-28mfh" pod "34d081d4-e311-40f4-9695-abcc8ff40605" (UID: "34d081d4-e311-40f4-9695-abcc8ff40605"). InnerVolumeSpecName "kube-api-access-28mfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:29.495888 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.484042 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d5nj\" (UniqueName: \"kubernetes.io/projected/5dbf96e5-22cf-4e5b-8208-470f9c1f616c-kube-api-access-4d5nj\") pod \"payload-pre-processing-e2e-shared-b-w6-53dd05-5b8b7fd495-b6jn6\" (UID: \"5dbf96e5-22cf-4e5b-8208-470f9c1f616c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-53dd05-5b8b7fd495-b6jn6" Nov 01 22:56:29.495888 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.484138 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9tq4\" (UniqueName: \"kubernetes.io/projected/2c1fbfd9-c189-4b15-92aa-394d88d95ad8-kube-api-access-z9tq4\") pod \"payload-processing-e2e-shared-b-w6-53dd05-fd7f899cc-tctb8\" (UID: \"2c1fbfd9-c189-4b15-92aa-394d88d95ad8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-53dd05-fd7f899cc-tctb8" Nov 01 22:56:29.495888 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.484230 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dbf96e5-22cf-4e5b-8208-470f9c1f616c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-53dd05-5b8b7fd495-b6jn6\" (UID: \"5dbf96e5-22cf-4e5b-8208-470f9c1f616c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-53dd05-5b8b7fd495-b6jn6" Nov 01 22:56:29.495888 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.484442 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c1fbfd9-c189-4b15-92aa-394d88d95ad8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-53dd05-fd7f899cc-tctb8\" (UID: \"2c1fbfd9-c189-4b15-92aa-394d88d95ad8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-53dd05-fd7f899cc-tctb8" Nov 01 22:56:29.495888 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.491227 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dbf96e5-22cf-4e5b-8208-470f9c1f616c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-53dd05-5b8b7fd495-b6jn6\" (UID: \"5dbf96e5-22cf-4e5b-8208-470f9c1f616c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-53dd05-5b8b7fd495-b6jn6" Nov 01 22:56:29.495888 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.491272 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34d081d4-e311-40f4-9695-abcc8ff40605-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:29.495888 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.491302 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28mfh\" (UniqueName: \"kubernetes.io/projected/34d081d4-e311-40f4-9695-abcc8ff40605-kube-api-access-28mfh\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:29.495888 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.492765 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-istio-envoy" pod "34d081d4-e311-40f4-9695-abcc8ff40605" (UID: "34d081d4-e311-40f4-9695-abcc8ff40605"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:29.583855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.573623 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d5nj\" (UniqueName: \"kubernetes.io/projected/5dbf96e5-22cf-4e5b-8208-470f9c1f616c-kube-api-access-4d5nj\") pod \"payload-pre-processing-e2e-shared-b-w6-53dd05-5b8b7fd495-b6jn6\" (UID: \"5dbf96e5-22cf-4e5b-8208-470f9c1f616c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-53dd05-5b8b7fd495-b6jn6" Nov 01 22:56:29.616270 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.613865 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c1fbfd9-c189-4b15-92aa-394d88d95ad8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-53dd05-fd7f899cc-tctb8\" (UID: \"2c1fbfd9-c189-4b15-92aa-394d88d95ad8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-53dd05-fd7f899cc-tctb8" Nov 01 22:56:29.616270 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.614170 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9tq4\" (UniqueName: \"kubernetes.io/projected/2c1fbfd9-c189-4b15-92aa-394d88d95ad8-kube-api-access-z9tq4\") pod \"payload-processing-e2e-shared-b-w6-53dd05-fd7f899cc-tctb8\" (UID: \"2c1fbfd9-c189-4b15-92aa-394d88d95ad8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-53dd05-fd7f899cc-tctb8" Nov 01 22:56:29.616270 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.614349 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34d081d4-e311-40f4-9695-abcc8ff40605-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:29.616270 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.615628 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c1fbfd9-c189-4b15-92aa-394d88d95ad8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-53dd05-fd7f899cc-tctb8\" (UID: \"2c1fbfd9-c189-4b15-92aa-394d88d95ad8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-53dd05-fd7f899cc-tctb8" Nov 01 22:56:29.664911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.635609 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9tq4\" (UniqueName: \"kubernetes.io/projected/2c1fbfd9-c189-4b15-92aa-394d88d95ad8-kube-api-access-z9tq4\") pod \"payload-processing-e2e-shared-b-w6-53dd05-fd7f899cc-tctb8\" (UID: \"2c1fbfd9-c189-4b15-92aa-394d88d95ad8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-53dd05-fd7f899cc-tctb8" Nov 01 22:56:29.678886 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.669322 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-53dd05-fd7f899cc-tctb8" Nov 01 22:56:29.679889 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.670321 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-53dd05-fd7f899cc-tctb8"] Nov 01 22:56:29.688575 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.686667 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" event={"ID":"665520e7-d3f3-4dd9-a7a5-3992fd827544","Type":"ContainerStarted","Data":"cbd00e2dc954916bb266d48e01ba6febbf7539e77742536a71c145f8147aa0d9"} Nov 01 22:56:29.696604 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.696554 5011 generic.go:356] "Generic (PLEG): container finished" podID="34d081d4-e311-40f4-9695-abcc8ff40605" containerID="ed3342ce267adabf14d5b803ff51e867d579d96a81c437122b619fdbc86c7428" exitCode=0 Nov 01 22:56:29.696807 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.696617 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" event={"ID":"34d081d4-e311-40f4-9695-abcc8ff40605","Type":"ContainerDied","Data":"ed3342ce267adabf14d5b803ff51e867d579d96a81c437122b619fdbc86c7428"} Nov 01 22:56:29.696807 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.696666 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" event={"ID":"34d081d4-e311-40f4-9695-abcc8ff40605","Type":"ContainerDied","Data":"f7feafe8fee61bbb92f9f312a7ec520d6ba5969cb7fd63e7b7c532b9d6638109"} Nov 01 22:56:29.696807 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.696696 5011 scope.go:122] "RemoveContainer" containerID="ed3342ce267adabf14d5b803ff51e867d579d96a81c437122b619fdbc86c7428" Nov 01 22:56:29.697124 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.696906 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh" Nov 01 22:56:29.747328 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.747305 5011 scope.go:122] "RemoveContainer" containerID="ed3342ce267adabf14d5b803ff51e867d579d96a81c437122b619fdbc86c7428" Nov 01 22:56:29.752112 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:56:29.750977 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed3342ce267adabf14d5b803ff51e867d579d96a81c437122b619fdbc86c7428\": container with ID starting with ed3342ce267adabf14d5b803ff51e867d579d96a81c437122b619fdbc86c7428 not found: ID does not exist" containerID="ed3342ce267adabf14d5b803ff51e867d579d96a81c437122b619fdbc86c7428" Nov 01 22:56:29.752112 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.751019 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed3342ce267adabf14d5b803ff51e867d579d96a81c437122b619fdbc86c7428"} err="failed to get container status \"ed3342ce267adabf14d5b803ff51e867d579d96a81c437122b619fdbc86c7428\": rpc error: code = NotFound desc = could not find container \"ed3342ce267adabf14d5b803ff51e867d579d96a81c437122b619fdbc86c7428\": container with ID starting with ed3342ce267adabf14d5b803ff51e867d579d96a81c437122b619fdbc86c7428 not found: ID does not exist" Nov 01 22:56:29.775167 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.775140 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh"] Nov 01 22:56:29.782928 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.782900 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-zjhrh"] Nov 01 22:56:29.784086 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.783315 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-53dd05-5b8b7fd495-b6jn6" Nov 01 22:56:29.869022 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.868947 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:56:29.871672 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.871389 5011 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 01 22:56:29.871672 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.871433 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" podUID="665520e7-d3f3-4dd9-a7a5-3992fd827544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 01 22:56:29.996702 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:56:29.996646 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c1fbfd9_c189_4b15_92aa_394d88d95ad8.slice/crio-6310721042c813c4b126e7c1dbd5fe7f9d7e75bb20e9c1cda5d56cd6f9119ed9 WatchSource:0}: Error finding container 6310721042c813c4b126e7c1dbd5fe7f9d7e75bb20e9c1cda5d56cd6f9119ed9: Status 404 returned error can't find the container with id 6310721042c813c4b126e7c1dbd5fe7f9d7e75bb20e9c1cda5d56cd6f9119ed9 Nov 01 22:56:29.997404 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:29.997158 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-53dd05-fd7f899cc-tctb8"] Nov 01 22:56:30.104381 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:30.104343 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-53dd05-5b8b7fd495-b6jn6"] Nov 01 22:56:30.709351 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:30.709312 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-53dd05-fd7f899cc-tctb8" event={"ID":"2c1fbfd9-c189-4b15-92aa-394d88d95ad8","Type":"ContainerStarted","Data":"51c3422849600f7161e0bc6dc740a78a912a556bedd1e67aad934bd93a75db08"} Nov 01 22:56:30.709572 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:30.709382 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-53dd05-fd7f899cc-tctb8" Nov 01 22:56:30.709572 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:30.709400 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-53dd05-fd7f899cc-tctb8" event={"ID":"2c1fbfd9-c189-4b15-92aa-394d88d95ad8","Type":"ContainerStarted","Data":"6310721042c813c4b126e7c1dbd5fe7f9d7e75bb20e9c1cda5d56cd6f9119ed9"} Nov 01 22:56:30.712681 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:30.712653 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-53dd05-5b8b7fd495-b6jn6" event={"ID":"5dbf96e5-22cf-4e5b-8208-470f9c1f616c","Type":"ContainerStarted","Data":"98e61759105f34f7a88c036dd865de0a554250924e18b91157cef2f40254f950"} Nov 01 22:56:30.712846 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:30.712698 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-53dd05-5b8b7fd495-b6jn6" event={"ID":"5dbf96e5-22cf-4e5b-8208-470f9c1f616c","Type":"ContainerStarted","Data":"c94e168859cb936f18ebc57bc093b65856762ce04683714233ab17a896e042ec"} Nov 01 22:56:30.712846 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:30.712729 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-53dd05-5b8b7fd495-b6jn6" Nov 01 22:56:30.727817 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:30.727768 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-53dd05-fd7f899cc-tctb8" podStartSLOduration=1.72775477 podStartE2EDuration="1.72775477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:56:30.726497701 +0000 UTC m=+1959.131088350" watchObservedRunningTime="2026-11-01 22:56:30.72775477 +0000 UTC m=+1959.132345401" Nov 01 22:56:30.774643 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:30.774575 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-53dd05-5b8b7fd495-b6jn6" podStartSLOduration=1.774561135 podStartE2EDuration="1.774561135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:56:30.769953743 +0000 UTC m=+1959.174544378" watchObservedRunningTime="2026-11-01 22:56:30.774561135 +0000 UTC m=+1959.179151767" Nov 01 22:56:30.891745 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:30.891421 5011 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 01 22:56:30.891745 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:30.891486 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" podUID="665520e7-d3f3-4dd9-a7a5-3992fd827544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 01 22:56:31.774420 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:31.774387 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34d081d4-e311-40f4-9695-abcc8ff40605" path="/var/lib/kubelet/pods/34d081d4-e311-40f4-9695-abcc8ff40605/volumes" Nov 01 22:56:31.869795 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:31.869757 5011 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 01 22:56:31.870048 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:31.869813 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" podUID="665520e7-d3f3-4dd9-a7a5-3992fd827544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 01 22:56:32.869854 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:32.869476 5011 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 01 22:56:32.869854 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:32.869564 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" podUID="665520e7-d3f3-4dd9-a7a5-3992fd827544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 01 22:56:33.100053 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:33.099999 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e716d-cbd5484dc-t8bxj" podUID="8d0fc024-7f80-4ca6-a234-3210cee311e4" containerName="payload-pre-processing" containerID="cri-o://8071f887dcd95d13880e6dfcfe45248caf33924a778f5fdab44680ce9ed00825" gracePeriod=25 Nov 01 22:56:33.878584 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:33.878172 5011 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 01 22:56:33.878584 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:33.878231 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" podUID="665520e7-d3f3-4dd9-a7a5-3992fd827544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 01 22:56:33.907344 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:33.907315 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f07902-558cf9ffcf-nxkzp"] Nov 01 22:56:34.869072 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:34.868924 5011 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 01 22:56:34.869072 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:34.868997 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" podUID="665520e7-d3f3-4dd9-a7a5-3992fd827544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 01 22:56:35.869234 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:35.869197 5011 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 01 22:56:35.869697 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:35.869252 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" podUID="665520e7-d3f3-4dd9-a7a5-3992fd827544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 01 22:56:36.874589 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:36.874558 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:56:36.875124 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:36.874948 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:56:36.875667 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:36.875646 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 22:56:38.908374 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:38.908333 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f07902-558cf9ffcf-nxkzp" podUID="01447e5b-5c6c-48b1-a455-765af224343f" containerName="payload-pre-processing" containerID="cri-o://1c253276b4586b6b4d42b5633d9cf55921ce6389b9864df8025860fef89b837f" gracePeriod=25 Nov 01 22:56:39.079817 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:39.079779 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e716d-cbd5484dc-t8bxj" podUID="8d0fc024-7f80-4ca6-a234-3210cee311e4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:56:41.725287 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:41.725256 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-53dd05-fd7f899cc-tctb8" Nov 01 22:56:41.725709 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:41.725340 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-53dd05-5b8b7fd495-b6jn6" Nov 01 22:56:42.424428 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:42.424389 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f07902-558cf9ffcf-nxkzp" podUID="01447e5b-5c6c-48b1-a455-765af224343f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:56:44.021705 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:44.021665 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn"] Nov 01 22:56:44.022222 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:44.021952 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn" podUID="0a061f44-9252-4c10-9677-912c562c47ad" containerName="maas-api" containerID="cri-o://00e8386df3c8f82ff3b2a2ee9fc295de50367f9657fcba9aa53cbc8d146ecc38" gracePeriod=30 Nov 01 22:56:44.725778 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:44.725752 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn" Nov 01 22:56:44.783596 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:44.783560 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a061f44-9252-4c10-9677-912c562c47ad-kube-api-access-4mlvn\" (UniqueName: \"kubernetes.io/projected/0a061f44-9252-4c10-9677-912c562c47ad-kube-api-access-4mlvn\") pod \"0a061f44-9252-4c10-9677-912c562c47ad\" (UID: \"0a061f44-9252-4c10-9677-912c562c47ad\") " Nov 01 22:56:44.831655 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:44.831620 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a061f44-9252-4c10-9677-912c562c47ad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a061f44-9252-4c10-9677-912c562c47ad-metrics-certs\") pod \"0a061f44-9252-4c10-9677-912c562c47ad\" (UID: \"0a061f44-9252-4c10-9677-912c562c47ad\") " Nov 01 22:56:44.862855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:44.860247 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a061f44-9252-4c10-9677-912c562c47ad-kube-api-access-4mlvn" pod "0a061f44-9252-4c10-9677-912c562c47ad" (UID: "0a061f44-9252-4c10-9677-912c562c47ad"). InnerVolumeSpecName "kube-api-access-4mlvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:44.862855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:44.860566 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a061f44-9252-4c10-9677-912c562c47ad-metrics-certs" pod "0a061f44-9252-4c10-9677-912c562c47ad" (UID: "0a061f44-9252-4c10-9677-912c562c47ad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:56:44.921872 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:44.921803 5011 generic.go:356] "Generic (PLEG): container finished" podID="0a061f44-9252-4c10-9677-912c562c47ad" containerID="00e8386df3c8f82ff3b2a2ee9fc295de50367f9657fcba9aa53cbc8d146ecc38" exitCode=0 Nov 01 22:56:44.921872 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:44.921864 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn" event={"ID":"0a061f44-9252-4c10-9677-912c562c47ad","Type":"ContainerDied","Data":"00e8386df3c8f82ff3b2a2ee9fc295de50367f9657fcba9aa53cbc8d146ecc38"} Nov 01 22:56:44.922111 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:44.921915 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn" event={"ID":"0a061f44-9252-4c10-9677-912c562c47ad","Type":"ContainerDied","Data":"72a04f9c0fdf8c77acdb1d84f029e42fa5549dee2c09fed88849f3f56f201fc6"} Nov 01 22:56:44.922111 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:44.921935 5011 scope.go:122] "RemoveContainer" containerID="00e8386df3c8f82ff3b2a2ee9fc295de50367f9657fcba9aa53cbc8d146ecc38" Nov 01 22:56:44.922111 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:44.921873 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn" Nov 01 22:56:44.934218 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:44.934194 5011 scope.go:122] "RemoveContainer" containerID="00e8386df3c8f82ff3b2a2ee9fc295de50367f9657fcba9aa53cbc8d146ecc38" Nov 01 22:56:44.934519 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:56:44.934493 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00e8386df3c8f82ff3b2a2ee9fc295de50367f9657fcba9aa53cbc8d146ecc38\": container with ID starting with 00e8386df3c8f82ff3b2a2ee9fc295de50367f9657fcba9aa53cbc8d146ecc38 not found: ID does not exist" containerID="00e8386df3c8f82ff3b2a2ee9fc295de50367f9657fcba9aa53cbc8d146ecc38" Nov 01 22:56:44.934622 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:44.934531 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00e8386df3c8f82ff3b2a2ee9fc295de50367f9657fcba9aa53cbc8d146ecc38"} err="failed to get container status \"00e8386df3c8f82ff3b2a2ee9fc295de50367f9657fcba9aa53cbc8d146ecc38\": rpc error: code = NotFound desc = could not find container \"00e8386df3c8f82ff3b2a2ee9fc295de50367f9657fcba9aa53cbc8d146ecc38\": container with ID starting with 00e8386df3c8f82ff3b2a2ee9fc295de50367f9657fcba9aa53cbc8d146ecc38 not found: ID does not exist" Nov 01 22:56:44.953433 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:44.953374 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a061f44-9252-4c10-9677-912c562c47ad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a061f44-9252-4c10-9677-912c562c47ad-maas-api-tls\") pod \"0a061f44-9252-4c10-9677-912c562c47ad\" (UID: \"0a061f44-9252-4c10-9677-912c562c47ad\") " Nov 01 22:56:44.953558 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:44.953456 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a061f44-9252-4c10-9677-912c562c47ad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a061f44-9252-4c10-9677-912c562c47ad-openshift-service-ca\") pod \"0a061f44-9252-4c10-9677-912c562c47ad\" (UID: \"0a061f44-9252-4c10-9677-912c562c47ad\") " Nov 01 22:56:44.953769 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:44.953748 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a061f44-9252-4c10-9677-912c562c47ad-openshift-service-ca" pod "0a061f44-9252-4c10-9677-912c562c47ad" (UID: "0a061f44-9252-4c10-9677-912c562c47ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:56:44.954227 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:44.954208 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a061f44-9252-4c10-9677-912c562c47ad-openshift-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:44.954312 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:44.954232 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mlvn\" (UniqueName: \"kubernetes.io/projected/0a061f44-9252-4c10-9677-912c562c47ad-kube-api-access-4mlvn\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:44.954312 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:44.954248 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a061f44-9252-4c10-9677-912c562c47ad-metrics-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:44.955657 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:44.955638 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a061f44-9252-4c10-9677-912c562c47ad-maas-api-tls" pod "0a061f44-9252-4c10-9677-912c562c47ad" (UID: "0a061f44-9252-4c10-9677-912c562c47ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:56:45.055359 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:45.055315 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a061f44-9252-4c10-9677-912c562c47ad-maas-api-tls\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:45.252241 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:45.252207 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn"] Nov 01 22:56:45.259187 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:45.259159 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn"] Nov 01 22:56:45.757746 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:45.757524 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a061f44-9252-4c10-9677-912c562c47ad" path="/var/lib/kubelet/pods/0a061f44-9252-4c10-9677-912c562c47ad/volumes" Nov 01 22:56:46.964443 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:56:46.964411 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:56:47.663422 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:47.663373 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e716d-7f6bd8b76b-2f2rn" podUID="0a061f44-9252-4c10-9677-912c562c47ad" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.138:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:49.080081 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:49.080040 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e716d-cbd5484dc-t8bxj" podUID="8d0fc024-7f80-4ca6-a234-3210cee311e4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:56:49.606610 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:49.606560 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt"] Nov 01 22:56:49.608015 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:49.607992 5011 cpu_manager.go:414] "containerMap: removing container" podUID="0a061f44-9252-4c10-9677-912c562c47ad" containerName="maas-api" Nov 01 22:56:49.608190 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:49.608174 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a061f44-9252-4c10-9677-912c562c47ad" containerName="maas-api" Nov 01 22:56:49.608494 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:49.608468 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a061f44-9252-4c10-9677-912c562c47ad" containerName="maas-api" Nov 01 22:56:49.614723 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:49.614605 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt" Nov 01 22:56:49.633140 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:49.633107 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-53dd05-serving-cert\"" Nov 01 22:56:49.633986 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:49.633958 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt"] Nov 01 22:56:49.766284 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:49.766245 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92250b8d-b598-489f-a5d8-31a2d2ed954e-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt\" (UID: \"92250b8d-b598-489f-a5d8-31a2d2ed954e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt" Nov 01 22:56:49.766518 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:49.766502 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92250b8d-b598-489f-a5d8-31a2d2ed954e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt\" (UID: \"92250b8d-b598-489f-a5d8-31a2d2ed954e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt" Nov 01 22:56:49.766623 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:49.766601 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92250b8d-b598-489f-a5d8-31a2d2ed954e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt\" (UID: \"92250b8d-b598-489f-a5d8-31a2d2ed954e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt" Nov 01 22:56:49.766708 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:49.766696 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x46z\" (UniqueName: \"kubernetes.io/projected/92250b8d-b598-489f-a5d8-31a2d2ed954e-kube-api-access-2x46z\") pod \"maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt\" (UID: \"92250b8d-b598-489f-a5d8-31a2d2ed954e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt" Nov 01 22:56:49.871701 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:49.871573 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x46z\" (UniqueName: \"kubernetes.io/projected/92250b8d-b598-489f-a5d8-31a2d2ed954e-kube-api-access-2x46z\") pod \"maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt\" (UID: \"92250b8d-b598-489f-a5d8-31a2d2ed954e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt" Nov 01 22:56:49.871911 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:49.871778 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92250b8d-b598-489f-a5d8-31a2d2ed954e-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt\" (UID: \"92250b8d-b598-489f-a5d8-31a2d2ed954e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt" Nov 01 22:56:49.907860 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:49.898710 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92250b8d-b598-489f-a5d8-31a2d2ed954e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt\" (UID: \"92250b8d-b598-489f-a5d8-31a2d2ed954e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt" Nov 01 22:56:49.908323 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:49.908293 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92250b8d-b598-489f-a5d8-31a2d2ed954e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt\" (UID: \"92250b8d-b598-489f-a5d8-31a2d2ed954e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt" Nov 01 22:56:49.924858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:49.909426 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92250b8d-b598-489f-a5d8-31a2d2ed954e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt\" (UID: \"92250b8d-b598-489f-a5d8-31a2d2ed954e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt" Nov 01 22:56:49.925135 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:49.913504 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92250b8d-b598-489f-a5d8-31a2d2ed954e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt\" (UID: \"92250b8d-b598-489f-a5d8-31a2d2ed954e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt" Nov 01 22:56:49.954878 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:49.954824 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92250b8d-b598-489f-a5d8-31a2d2ed954e-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt\" (UID: \"92250b8d-b598-489f-a5d8-31a2d2ed954e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt" Nov 01 22:56:49.963040 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:49.958899 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x46z\" (UniqueName: \"kubernetes.io/projected/92250b8d-b598-489f-a5d8-31a2d2ed954e-kube-api-access-2x46z\") pod \"maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt\" (UID: \"92250b8d-b598-489f-a5d8-31a2d2ed954e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt" Nov 01 22:56:49.978440 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:49.978166 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt" Nov 01 22:56:50.401794 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:56:50.401751 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92250b8d_b598_489f_a5d8_31a2d2ed954e.slice/crio-2ef8fdbb91100ff0c22c5eead3c5544cba0c49c69e650313df3b0f1448abad06 WatchSource:0}: Error finding container 2ef8fdbb91100ff0c22c5eead3c5544cba0c49c69e650313df3b0f1448abad06: Status 404 returned error can't find the container with id 2ef8fdbb91100ff0c22c5eead3c5544cba0c49c69e650313df3b0f1448abad06 Nov 01 22:56:50.437986 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:50.422892 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt"] Nov 01 22:56:51.026154 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:51.026123 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt" event={"ID":"92250b8d-b598-489f-a5d8-31a2d2ed954e","Type":"ContainerStarted","Data":"2ef8fdbb91100ff0c22c5eead3c5544cba0c49c69e650313df3b0f1448abad06"} Nov 01 22:56:52.037364 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:52.037321 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt" event={"ID":"92250b8d-b598-489f-a5d8-31a2d2ed954e","Type":"ContainerStarted","Data":"0f0f04d84a0fb7ac5fa88ffcaed4b747eb9d00d44eb2bca3179f29abce82eff1"} Nov 01 22:56:52.037824 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:52.037390 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt" Nov 01 22:56:52.070608 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:52.070533 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt" podStartSLOduration=2.5710258550000002 podStartE2EDuration="3.070513344s" totalImagesPullingTime="499.487489ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:56:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:56:52.061613903 +0000 UTC m=+1980.466204533" watchObservedRunningTime="2026-11-01 22:56:52.070513344 +0000 UTC m=+1980.475103982" Nov 01 22:56:52.419174 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:52.419090 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f07902-558cf9ffcf-nxkzp" podUID="01447e5b-5c6c-48b1-a455-765af224343f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:56:53.402213 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:53.402190 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e716d-cbd5484dc-t8bxj" Nov 01 22:56:53.440462 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:53.440420 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d0fc024-7f80-4ca6-a234-3210cee311e4-kube-api-access-fk4kr\" (UniqueName: \"kubernetes.io/projected/8d0fc024-7f80-4ca6-a234-3210cee311e4-kube-api-access-fk4kr\") pod \"8d0fc024-7f80-4ca6-a234-3210cee311e4\" (UID: \"8d0fc024-7f80-4ca6-a234-3210cee311e4\") " Nov 01 22:56:53.440641 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:53.440564 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d0fc024-7f80-4ca6-a234-3210cee311e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d0fc024-7f80-4ca6-a234-3210cee311e4-plugins-config-volume\") pod \"8d0fc024-7f80-4ca6-a234-3210cee311e4\" (UID: \"8d0fc024-7f80-4ca6-a234-3210cee311e4\") " Nov 01 22:56:53.442434 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:53.442032 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d0fc024-7f80-4ca6-a234-3210cee311e4-plugins-config-volume" pod "8d0fc024-7f80-4ca6-a234-3210cee311e4" (UID: "8d0fc024-7f80-4ca6-a234-3210cee311e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:56:53.445657 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:53.445612 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d0fc024-7f80-4ca6-a234-3210cee311e4-kube-api-access-fk4kr" pod "8d0fc024-7f80-4ca6-a234-3210cee311e4" (UID: "8d0fc024-7f80-4ca6-a234-3210cee311e4"). InnerVolumeSpecName "kube-api-access-fk4kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:53.543261 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:53.543226 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk4kr\" (UniqueName: \"kubernetes.io/projected/8d0fc024-7f80-4ca6-a234-3210cee311e4-kube-api-access-fk4kr\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:53.543261 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:53.543258 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d0fc024-7f80-4ca6-a234-3210cee311e4-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:53.959046 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:53.958218 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt"] Nov 01 22:56:53.959528 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:53.959487 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt" podUID="6708ae18-425b-46be-87c4-2b5023e9ba8e" containerName="maas-api" containerID="cri-o://62f1fce8c878fe46c1b4dad9765070fc3d0e0dc3395ddf7b3c891b62f728914c" gracePeriod=30 Nov 01 22:56:54.059322 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:54.059278 5011 generic.go:356] "Generic (PLEG): container finished" podID="8d0fc024-7f80-4ca6-a234-3210cee311e4" containerID="8071f887dcd95d13880e6dfcfe45248caf33924a778f5fdab44680ce9ed00825" exitCode=0 Nov 01 22:56:54.059473 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:54.059345 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e716d-cbd5484dc-t8bxj" Nov 01 22:56:54.059473 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:54.059368 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e716d-cbd5484dc-t8bxj" event={"ID":"8d0fc024-7f80-4ca6-a234-3210cee311e4","Type":"ContainerDied","Data":"8071f887dcd95d13880e6dfcfe45248caf33924a778f5fdab44680ce9ed00825"} Nov 01 22:56:54.059473 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:54.059440 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e716d-cbd5484dc-t8bxj" event={"ID":"8d0fc024-7f80-4ca6-a234-3210cee311e4","Type":"ContainerDied","Data":"f9e0d406fbd1a5e524ccda42b729120c6f810c9f8c3cee5bd205ff96cd7b00d9"} Nov 01 22:56:54.059473 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:54.059468 5011 scope.go:122] "RemoveContainer" containerID="8071f887dcd95d13880e6dfcfe45248caf33924a778f5fdab44680ce9ed00825" Nov 01 22:56:54.079138 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:54.079079 5011 scope.go:122] "RemoveContainer" containerID="8071f887dcd95d13880e6dfcfe45248caf33924a778f5fdab44680ce9ed00825" Nov 01 22:56:54.079441 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:56:54.079383 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8071f887dcd95d13880e6dfcfe45248caf33924a778f5fdab44680ce9ed00825\": container with ID starting with 8071f887dcd95d13880e6dfcfe45248caf33924a778f5fdab44680ce9ed00825 not found: ID does not exist" containerID="8071f887dcd95d13880e6dfcfe45248caf33924a778f5fdab44680ce9ed00825" Nov 01 22:56:54.079441 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:54.079423 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8071f887dcd95d13880e6dfcfe45248caf33924a778f5fdab44680ce9ed00825"} err="failed to get container status \"8071f887dcd95d13880e6dfcfe45248caf33924a778f5fdab44680ce9ed00825\": rpc error: code = NotFound desc = could not find container \"8071f887dcd95d13880e6dfcfe45248caf33924a778f5fdab44680ce9ed00825\": container with ID starting with 8071f887dcd95d13880e6dfcfe45248caf33924a778f5fdab44680ce9ed00825 not found: ID does not exist" Nov 01 22:56:54.081933 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:54.081908 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0e716d-cbd5484dc-t8bxj"] Nov 01 22:56:54.088274 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:54.088249 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0e716d-cbd5484dc-t8bxj"] Nov 01 22:56:54.708854 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:54.705676 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt" Nov 01 22:56:54.774862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:54.770081 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6708ae18-425b-46be-87c4-2b5023e9ba8e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6708ae18-425b-46be-87c4-2b5023e9ba8e-maas-api-tls\") pod \"6708ae18-425b-46be-87c4-2b5023e9ba8e\" (UID: \"6708ae18-425b-46be-87c4-2b5023e9ba8e\") " Nov 01 22:56:54.774862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:54.770139 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6708ae18-425b-46be-87c4-2b5023e9ba8e-kube-api-access-t7drb\" (UniqueName: \"kubernetes.io/projected/6708ae18-425b-46be-87c4-2b5023e9ba8e-kube-api-access-t7drb\") pod \"6708ae18-425b-46be-87c4-2b5023e9ba8e\" (UID: \"6708ae18-425b-46be-87c4-2b5023e9ba8e\") " Nov 01 22:56:54.774862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:54.770199 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6708ae18-425b-46be-87c4-2b5023e9ba8e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6708ae18-425b-46be-87c4-2b5023e9ba8e-metrics-certs\") pod \"6708ae18-425b-46be-87c4-2b5023e9ba8e\" (UID: \"6708ae18-425b-46be-87c4-2b5023e9ba8e\") " Nov 01 22:56:54.774862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:54.770408 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6708ae18-425b-46be-87c4-2b5023e9ba8e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6708ae18-425b-46be-87c4-2b5023e9ba8e-openshift-service-ca\") pod \"6708ae18-425b-46be-87c4-2b5023e9ba8e\" (UID: \"6708ae18-425b-46be-87c4-2b5023e9ba8e\") " Nov 01 22:56:54.779855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:54.776296 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6708ae18-425b-46be-87c4-2b5023e9ba8e-maas-api-tls" pod "6708ae18-425b-46be-87c4-2b5023e9ba8e" (UID: "6708ae18-425b-46be-87c4-2b5023e9ba8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:56:54.779855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:54.776766 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6708ae18-425b-46be-87c4-2b5023e9ba8e-metrics-certs" pod "6708ae18-425b-46be-87c4-2b5023e9ba8e" (UID: "6708ae18-425b-46be-87c4-2b5023e9ba8e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:56:54.795858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:54.792265 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6708ae18-425b-46be-87c4-2b5023e9ba8e-openshift-service-ca" pod "6708ae18-425b-46be-87c4-2b5023e9ba8e" (UID: "6708ae18-425b-46be-87c4-2b5023e9ba8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:56:54.849738 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:54.834060 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6708ae18-425b-46be-87c4-2b5023e9ba8e-kube-api-access-t7drb" pod "6708ae18-425b-46be-87c4-2b5023e9ba8e" (UID: "6708ae18-425b-46be-87c4-2b5023e9ba8e"). InnerVolumeSpecName "kube-api-access-t7drb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:54.872779 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:54.872690 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6708ae18-425b-46be-87c4-2b5023e9ba8e-openshift-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:54.872779 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:54.872730 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6708ae18-425b-46be-87c4-2b5023e9ba8e-maas-api-tls\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:54.872779 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:54.872747 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7drb\" (UniqueName: \"kubernetes.io/projected/6708ae18-425b-46be-87c4-2b5023e9ba8e-kube-api-access-t7drb\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:54.872779 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:54.872761 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6708ae18-425b-46be-87c4-2b5023e9ba8e-metrics-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:55.070634 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:55.070601 5011 generic.go:356] "Generic (PLEG): container finished" podID="6708ae18-425b-46be-87c4-2b5023e9ba8e" containerID="62f1fce8c878fe46c1b4dad9765070fc3d0e0dc3395ddf7b3c891b62f728914c" exitCode=0 Nov 01 22:56:55.070851 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:55.070642 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt" Nov 01 22:56:55.070851 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:55.070681 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt" event={"ID":"6708ae18-425b-46be-87c4-2b5023e9ba8e","Type":"ContainerDied","Data":"62f1fce8c878fe46c1b4dad9765070fc3d0e0dc3395ddf7b3c891b62f728914c"} Nov 01 22:56:55.070851 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:55.070738 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt" event={"ID":"6708ae18-425b-46be-87c4-2b5023e9ba8e","Type":"ContainerDied","Data":"4b7cecd7fd3720fb48ac4c35a5e74501d14908d14cc28c3851626bf95569a547"} Nov 01 22:56:55.070851 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:55.070766 5011 scope.go:122] "RemoveContainer" containerID="62f1fce8c878fe46c1b4dad9765070fc3d0e0dc3395ddf7b3c891b62f728914c" Nov 01 22:56:55.083045 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:55.083020 5011 scope.go:122] "RemoveContainer" containerID="62f1fce8c878fe46c1b4dad9765070fc3d0e0dc3395ddf7b3c891b62f728914c" Nov 01 22:56:55.083378 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:56:55.083357 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62f1fce8c878fe46c1b4dad9765070fc3d0e0dc3395ddf7b3c891b62f728914c\": container with ID starting with 62f1fce8c878fe46c1b4dad9765070fc3d0e0dc3395ddf7b3c891b62f728914c not found: ID does not exist" containerID="62f1fce8c878fe46c1b4dad9765070fc3d0e0dc3395ddf7b3c891b62f728914c" Nov 01 22:56:55.083446 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:55.083388 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62f1fce8c878fe46c1b4dad9765070fc3d0e0dc3395ddf7b3c891b62f728914c"} err="failed to get container status \"62f1fce8c878fe46c1b4dad9765070fc3d0e0dc3395ddf7b3c891b62f728914c\": rpc error: code = NotFound desc = could not find container \"62f1fce8c878fe46c1b4dad9765070fc3d0e0dc3395ddf7b3c891b62f728914c\": container with ID starting with 62f1fce8c878fe46c1b4dad9765070fc3d0e0dc3395ddf7b3c891b62f728914c not found: ID does not exist" Nov 01 22:56:55.096713 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:55.096679 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt"] Nov 01 22:56:55.101056 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:55.101032 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f07902-688ffc85f5-wfgxt"] Nov 01 22:56:55.749980 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:55.749936 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6708ae18-425b-46be-87c4-2b5023e9ba8e" path="/var/lib/kubelet/pods/6708ae18-425b-46be-87c4-2b5023e9ba8e/volumes" Nov 01 22:56:55.750583 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:55.750564 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d0fc024-7f80-4ca6-a234-3210cee311e4" path="/var/lib/kubelet/pods/8d0fc024-7f80-4ca6-a234-3210cee311e4/volumes" Nov 01 22:56:56.031262 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:56.031187 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7"] Nov 01 22:56:56.031417 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:56.031399 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" podUID="e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" containerName="istio-proxy" containerID="cri-o://82ea74e192e9f057cc53931fc4c8f2e06cb4dd85828862bb85b2802a78ac5170" gracePeriod=30 Nov 01 22:56:56.085901 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:56.085869 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr"] Nov 01 22:56:56.086069 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:56.086045 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" podUID="a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6" containerName="istio-proxy" containerID="cri-o://9d72fc67ed44c15059a2173bde04077e4daadd84aa24f1fe9dd1fb06154f7f8a" gracePeriod=30 Nov 01 22:56:58.082857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:58.075072 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt" Nov 01 22:56:58.417934 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:58.417820 5011 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:56:58.417934 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:58.417904 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" podUID="e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:56:59.129103 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:59.129070 5011 generic.go:356] "Generic (PLEG): container finished" podID="01447e5b-5c6c-48b1-a455-765af224343f" containerID="1c253276b4586b6b4d42b5633d9cf55921ce6389b9864df8025860fef89b837f" exitCode=0 Nov 01 22:56:59.129458 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:59.129139 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f07902-558cf9ffcf-nxkzp" event={"ID":"01447e5b-5c6c-48b1-a455-765af224343f","Type":"ContainerDied","Data":"1c253276b4586b6b4d42b5633d9cf55921ce6389b9864df8025860fef89b837f"} Nov 01 22:56:59.129458 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:59.129194 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f07902-558cf9ffcf-nxkzp" event={"ID":"01447e5b-5c6c-48b1-a455-765af224343f","Type":"ContainerDied","Data":"b7cb153666b58cda1f64d6bf80f9731395178b98e262213417c452d5064705d6"} Nov 01 22:56:59.129458 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:59.129210 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7cb153666b58cda1f64d6bf80f9731395178b98e262213417c452d5064705d6" Nov 01 22:56:59.148117 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:59.148085 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f07902-558cf9ffcf-nxkzp" Nov 01 22:56:59.281656 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:59.281562 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01447e5b-5c6c-48b1-a455-765af224343f-kube-api-access-cmntf\" (UniqueName: \"kubernetes.io/projected/01447e5b-5c6c-48b1-a455-765af224343f-kube-api-access-cmntf\") pod \"01447e5b-5c6c-48b1-a455-765af224343f\" (UID: \"01447e5b-5c6c-48b1-a455-765af224343f\") " Nov 01 22:56:59.281882 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:59.281768 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01447e5b-5c6c-48b1-a455-765af224343f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01447e5b-5c6c-48b1-a455-765af224343f-plugins-config-volume\") pod \"01447e5b-5c6c-48b1-a455-765af224343f\" (UID: \"01447e5b-5c6c-48b1-a455-765af224343f\") " Nov 01 22:56:59.282180 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:59.282156 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01447e5b-5c6c-48b1-a455-765af224343f-plugins-config-volume" pod "01447e5b-5c6c-48b1-a455-765af224343f" (UID: "01447e5b-5c6c-48b1-a455-765af224343f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:56:59.282452 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:59.282430 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01447e5b-5c6c-48b1-a455-765af224343f-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:59.284080 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:59.284057 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01447e5b-5c6c-48b1-a455-765af224343f-kube-api-access-cmntf" pod "01447e5b-5c6c-48b1-a455-765af224343f" (UID: "01447e5b-5c6c-48b1-a455-765af224343f"). InnerVolumeSpecName "kube-api-access-cmntf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:59.383893 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:56:59.383859 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmntf\" (UniqueName: \"kubernetes.io/projected/01447e5b-5c6c-48b1-a455-765af224343f-kube-api-access-cmntf\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:00.137313 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:00.137239 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f07902-558cf9ffcf-nxkzp" Nov 01 22:57:00.156228 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:00.156177 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f07902-558cf9ffcf-nxkzp"] Nov 01 22:57:00.160226 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:00.160197 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f07902-558cf9ffcf-nxkzp"] Nov 01 22:57:01.431123 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.431094 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:57:01.503024 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.503000 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:57:01.534225 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.534181 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-workload-certs\") pod \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " Nov 01 22:57:01.534368 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.534261 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-kube-api-access-9lr2r\" (UniqueName: \"kubernetes.io/projected/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-kube-api-access-9lr2r\") pod \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " Nov 01 22:57:01.534368 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.534337 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-workload-socket\") pod \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " Nov 01 22:57:01.534456 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.534383 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istio-podinfo\") pod \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " Nov 01 22:57:01.534513 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.534498 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istio-token\" (UniqueName: \"kubernetes.io/projected/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istio-token\") pod \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " Nov 01 22:57:01.534741 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.534723 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istio-envoy\") pod \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " Nov 01 22:57:01.534873 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.534856 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-credential-socket\") pod \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " Nov 01 22:57:01.534945 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.534917 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istiod-ca-cert\") pod \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " Nov 01 22:57:01.534945 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.534915 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-workload-socket" pod "e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" (UID: "e66d728f-81d0-4dd4-9d6d-1fe881bfbd54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:57:01.535062 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.534953 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istio-data\") pod \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\" (UID: \"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54\") " Nov 01 22:57:01.535173 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.535147 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-credential-socket" pod "e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" (UID: "e66d728f-81d0-4dd4-9d6d-1fe881bfbd54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:57:01.535288 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.535258 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-workload-certs" pod "e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" (UID: "e66d728f-81d0-4dd4-9d6d-1fe881bfbd54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:57:01.535412 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.535390 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istio-data" pod "e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" (UID: "e66d728f-81d0-4dd4-9d6d-1fe881bfbd54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:57:01.535934 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.535910 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:01.536036 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.535936 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:01.536036 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.535952 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:01.536036 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.535964 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:01.536354 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.535598 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istiod-ca-cert" pod "e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" (UID: "e66d728f-81d0-4dd4-9d6d-1fe881bfbd54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:57:01.537935 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.537910 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istio-envoy" pod "e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" (UID: "e66d728f-81d0-4dd4-9d6d-1fe881bfbd54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:57:01.538132 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.538109 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-kube-api-access-9lr2r" pod "e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" (UID: "e66d728f-81d0-4dd4-9d6d-1fe881bfbd54"). InnerVolumeSpecName "kube-api-access-9lr2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:57:01.538311 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.538291 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istio-podinfo" pod "e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" (UID: "e66d728f-81d0-4dd4-9d6d-1fe881bfbd54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:57:01.539281 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.539263 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istio-token" pod "e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" (UID: "e66d728f-81d0-4dd4-9d6d-1fe881bfbd54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:57:01.648754 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.639742 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istiod-ca-cert\") pod \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " Nov 01 22:57:01.648754 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.639878 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istio-token\" (UniqueName: \"kubernetes.io/projected/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istio-token\") pod \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " Nov 01 22:57:01.648754 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.639943 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istio-envoy\") pod \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " Nov 01 22:57:01.648754 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.639975 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-workload-socket\") pod \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " Nov 01 22:57:01.648754 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.640028 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-workload-certs\") pod \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " Nov 01 22:57:01.648754 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.640207 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-kube-api-access-8dfl9\" (UniqueName: \"kubernetes.io/projected/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-kube-api-access-8dfl9\") pod \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " Nov 01 22:57:01.648754 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.640278 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istio-data\") pod \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " Nov 01 22:57:01.648754 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.640326 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istio-podinfo\") pod \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " Nov 01 22:57:01.648754 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.640407 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-credential-socket\") pod \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\" (UID: \"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6\") " Nov 01 22:57:01.649740 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.649702 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-workload-socket" pod "a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6" (UID: "a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:57:01.649740 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.649722 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:01.650006 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.649771 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:01.650006 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.649789 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:01.650006 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.649805 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lr2r\" (UniqueName: \"kubernetes.io/projected/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-kube-api-access-9lr2r\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:01.650006 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.649819 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:01.652118 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.652075 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istiod-ca-cert" pod "a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6" (UID: "a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:57:01.652366 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.652130 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-workload-certs" pod "a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6" (UID: "a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:57:01.652584 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.652494 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istio-data" pod "a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6" (UID: "a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:57:01.652824 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.652799 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-credential-socket" pod "a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6" (UID: "a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:57:01.652824 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.652804 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istio-token" pod "a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6" (UID: "a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:57:01.660226 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.660161 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istio-envoy" pod "a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6" (UID: "a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:57:01.663152 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.663123 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istio-podinfo" pod "a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6" (UID: "a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:57:01.671354 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.671017 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-kube-api-access-8dfl9" pod "a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6" (UID: "a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6"). InnerVolumeSpecName "kube-api-access-8dfl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:57:01.751247 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.751207 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dfl9\" (UniqueName: \"kubernetes.io/projected/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-kube-api-access-8dfl9\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:01.751247 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.751245 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:01.751455 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.751260 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:01.751455 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.751275 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:01.751455 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.751288 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:01.751455 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.751303 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:01.751455 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.751315 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:01.751455 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.751328 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:01.751455 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.751342 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:01.752702 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:01.752680 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01447e5b-5c6c-48b1-a455-765af224343f" path="/var/lib/kubelet/pods/01447e5b-5c6c-48b1-a455-765af224343f/volumes" Nov 01 22:57:02.281855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:02.273414 5011 generic.go:356] "Generic (PLEG): container finished" podID="e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" containerID="82ea74e192e9f057cc53931fc4c8f2e06cb4dd85828862bb85b2802a78ac5170" exitCode=0 Nov 01 22:57:02.281855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:02.273610 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" event={"ID":"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54","Type":"ContainerDied","Data":"82ea74e192e9f057cc53931fc4c8f2e06cb4dd85828862bb85b2802a78ac5170"} Nov 01 22:57:02.281855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:02.273668 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" event={"ID":"e66d728f-81d0-4dd4-9d6d-1fe881bfbd54","Type":"ContainerDied","Data":"73c13b0dbbd6b00d87081d85e095fc9351fa2bec1d551f6d50a8fb5cbcdb1efb"} Nov 01 22:57:02.281855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:02.274509 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7" Nov 01 22:57:02.281855 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:02.277086 5011 scope.go:122] "RemoveContainer" containerID="82ea74e192e9f057cc53931fc4c8f2e06cb4dd85828862bb85b2802a78ac5170" Nov 01 22:57:02.290263 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:02.288542 5011 generic.go:356] "Generic (PLEG): container finished" podID="a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6" containerID="9d72fc67ed44c15059a2173bde04077e4daadd84aa24f1fe9dd1fb06154f7f8a" exitCode=0 Nov 01 22:57:02.290263 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:02.288772 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" event={"ID":"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6","Type":"ContainerDied","Data":"9d72fc67ed44c15059a2173bde04077e4daadd84aa24f1fe9dd1fb06154f7f8a"} Nov 01 22:57:02.290263 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:02.288822 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" event={"ID":"a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6","Type":"ContainerDied","Data":"798cd7c221dc74648db6dc509de988af50dadc8670dca0da30eb065fde095982"} Nov 01 22:57:02.290263 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:02.288941 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr" Nov 01 22:57:02.314820 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:02.314468 5011 scope.go:122] "RemoveContainer" containerID="82ea74e192e9f057cc53931fc4c8f2e06cb4dd85828862bb85b2802a78ac5170" Nov 01 22:57:02.319187 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:57:02.317324 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82ea74e192e9f057cc53931fc4c8f2e06cb4dd85828862bb85b2802a78ac5170\": container with ID starting with 82ea74e192e9f057cc53931fc4c8f2e06cb4dd85828862bb85b2802a78ac5170 not found: ID does not exist" containerID="82ea74e192e9f057cc53931fc4c8f2e06cb4dd85828862bb85b2802a78ac5170" Nov 01 22:57:02.319187 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:02.317364 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82ea74e192e9f057cc53931fc4c8f2e06cb4dd85828862bb85b2802a78ac5170"} err="failed to get container status \"82ea74e192e9f057cc53931fc4c8f2e06cb4dd85828862bb85b2802a78ac5170\": rpc error: code = NotFound desc = could not find container \"82ea74e192e9f057cc53931fc4c8f2e06cb4dd85828862bb85b2802a78ac5170\": container with ID starting with 82ea74e192e9f057cc53931fc4c8f2e06cb4dd85828862bb85b2802a78ac5170 not found: ID does not exist" Nov 01 22:57:02.319187 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:02.317389 5011 scope.go:122] "RemoveContainer" containerID="9d72fc67ed44c15059a2173bde04077e4daadd84aa24f1fe9dd1fb06154f7f8a" Nov 01 22:57:02.345077 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:02.345046 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7"] Nov 01 22:57:02.364292 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:02.364268 5011 scope.go:122] "RemoveContainer" containerID="9d72fc67ed44c15059a2173bde04077e4daadd84aa24f1fe9dd1fb06154f7f8a" Nov 01 22:57:02.364807 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:57:02.364771 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d72fc67ed44c15059a2173bde04077e4daadd84aa24f1fe9dd1fb06154f7f8a\": container with ID starting with 9d72fc67ed44c15059a2173bde04077e4daadd84aa24f1fe9dd1fb06154f7f8a not found: ID does not exist" containerID="9d72fc67ed44c15059a2173bde04077e4daadd84aa24f1fe9dd1fb06154f7f8a" Nov 01 22:57:02.365272 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:02.364816 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d72fc67ed44c15059a2173bde04077e4daadd84aa24f1fe9dd1fb06154f7f8a"} err="failed to get container status \"9d72fc67ed44c15059a2173bde04077e4daadd84aa24f1fe9dd1fb06154f7f8a\": rpc error: code = NotFound desc = could not find container \"9d72fc67ed44c15059a2173bde04077e4daadd84aa24f1fe9dd1fb06154f7f8a\": container with ID starting with 9d72fc67ed44c15059a2173bde04077e4daadd84aa24f1fe9dd1fb06154f7f8a not found: ID does not exist" Nov 01 22:57:02.376870 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:02.376149 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-w26n7"] Nov 01 22:57:02.396737 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:02.396665 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr"] Nov 01 22:57:02.401734 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:02.401697 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5cscr"] Nov 01 22:57:03.263964 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.263901 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj"] Nov 01 22:57:03.265057 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.265029 5011 cpu_manager.go:414] "containerMap: removing container" podUID="6708ae18-425b-46be-87c4-2b5023e9ba8e" containerName="maas-api" Nov 01 22:57:03.265057 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.265057 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6708ae18-425b-46be-87c4-2b5023e9ba8e" containerName="maas-api" Nov 01 22:57:03.265409 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.265390 5011 cpu_manager.go:414] "containerMap: removing container" podUID="01447e5b-5c6c-48b1-a455-765af224343f" containerName="payload-pre-processing" Nov 01 22:57:03.265482 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.265412 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01447e5b-5c6c-48b1-a455-765af224343f" containerName="payload-pre-processing" Nov 01 22:57:03.265482 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.265430 5011 cpu_manager.go:414] "containerMap: removing container" podUID="e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" containerName="istio-proxy" Nov 01 22:57:03.265482 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.265452 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" containerName="istio-proxy" Nov 01 22:57:03.265482 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.265473 5011 cpu_manager.go:414] "containerMap: removing container" podUID="a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6" containerName="istio-proxy" Nov 01 22:57:03.265482 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.265481 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6" containerName="istio-proxy" Nov 01 22:57:03.265738 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.265497 5011 cpu_manager.go:414] "containerMap: removing container" podUID="8d0fc024-7f80-4ca6-a234-3210cee311e4" containerName="payload-pre-processing" Nov 01 22:57:03.265738 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.265504 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d0fc024-7f80-4ca6-a234-3210cee311e4" containerName="payload-pre-processing" Nov 01 22:57:03.265983 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.265964 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="6708ae18-425b-46be-87c4-2b5023e9ba8e" containerName="maas-api" Nov 01 22:57:03.266045 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.266000 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="01447e5b-5c6c-48b1-a455-765af224343f" containerName="payload-pre-processing" Nov 01 22:57:03.266045 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.266013 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" containerName="istio-proxy" Nov 01 22:57:03.266045 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.266028 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6" containerName="istio-proxy" Nov 01 22:57:03.266045 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.266039 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d0fc024-7f80-4ca6-a234-3210cee311e4" containerName="payload-pre-processing" Nov 01 22:57:03.275477 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.275454 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:03.275888 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.275863 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj"] Nov 01 22:57:03.283490 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.283465 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-788d7b-openshift-default-dockercfg-8d6wp\"" Nov 01 22:57:03.437867 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.432574 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03efd24-da7e-407d-896d-d22e46206274-istio-podinfo\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:03.450869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.447250 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-istio-envoy\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:03.450869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.447331 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-istio-data\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:03.450869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.447391 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5zch\" (UniqueName: \"kubernetes.io/projected/a03efd24-da7e-407d-896d-d22e46206274-kube-api-access-t5zch\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:03.450869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.447465 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a03efd24-da7e-407d-896d-d22e46206274-istio-token\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:03.450869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.447525 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-credential-socket\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:03.450869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.447614 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-workload-socket\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:03.450869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.447639 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03efd24-da7e-407d-896d-d22e46206274-istiod-ca-cert\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:03.450869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.447904 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-workload-certs\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:03.575914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.574937 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a03efd24-da7e-407d-896d-d22e46206274-istio-token\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:03.575914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.575025 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-credential-socket\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:03.575914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.575101 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-workload-socket\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:03.575914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.575125 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03efd24-da7e-407d-896d-d22e46206274-istiod-ca-cert\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:03.575914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.575299 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-workload-certs\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:03.575914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.575389 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03efd24-da7e-407d-896d-d22e46206274-istio-podinfo\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:03.575914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.575472 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-istio-envoy\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:03.575914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.575513 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-istio-data\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:03.575914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.575559 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5zch\" (UniqueName: \"kubernetes.io/projected/a03efd24-da7e-407d-896d-d22e46206274-kube-api-access-t5zch\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:03.577859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.576466 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-credential-socket\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:03.577859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.576710 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-workload-socket\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:03.577859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.577392 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03efd24-da7e-407d-896d-d22e46206274-istiod-ca-cert\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:03.577859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.577627 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-workload-certs\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:03.590862 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.578496 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-istio-data\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:03.601871 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.601460 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a03efd24-da7e-407d-896d-d22e46206274-istio-token\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:03.608864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.606331 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-istio-envoy\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:03.638521 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.634437 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03efd24-da7e-407d-896d-d22e46206274-istio-podinfo\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:03.638521 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.635529 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5zch\" (UniqueName: \"kubernetes.io/projected/a03efd24-da7e-407d-896d-d22e46206274-kube-api-access-t5zch\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:03.751817 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.751787 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6" path="/var/lib/kubelet/pods/a83c24a9-8ffb-4b1b-910a-c7c9b0fadef6/volumes" Nov 01 22:57:03.752483 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.752465 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e66d728f-81d0-4dd4-9d6d-1fe881bfbd54" path="/var/lib/kubelet/pods/e66d728f-81d0-4dd4-9d6d-1fe881bfbd54/volumes" Nov 01 22:57:03.905582 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:03.905495 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:04.296290 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:04.296258 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj"] Nov 01 22:57:04.296636 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:57:04.296301 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda03efd24_da7e_407d_896d_d22e46206274.slice/crio-04320d38c6c78a09141b44cbb4bdbe3243e5ae5d65f401d267c71936785d3589 WatchSource:0}: Error finding container 04320d38c6c78a09141b44cbb4bdbe3243e5ae5d65f401d267c71936785d3589: Status 404 returned error can't find the container with id 04320d38c6c78a09141b44cbb4bdbe3243e5ae5d65f401d267c71936785d3589 Nov 01 22:57:04.299536 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:04.299501 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:57:04.299636 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:04.299571 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:57:04.299636 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:04.299601 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:57:04.325097 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:04.325061 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" event={"ID":"a03efd24-da7e-407d-896d-d22e46206274","Type":"ContainerStarted","Data":"04320d38c6c78a09141b44cbb4bdbe3243e5ae5d65f401d267c71936785d3589"} Nov 01 22:57:05.335780 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:05.335747 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" event={"ID":"a03efd24-da7e-407d-896d-d22e46206274","Type":"ContainerStarted","Data":"53a9b720e37ce41acd3fb311faa168f34d9b506fa7322cbe682798a67b036677"} Nov 01 22:57:05.361952 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:05.361884 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" podStartSLOduration=2.361865628 podStartE2EDuration="2.361865628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:57:05.357980475 +0000 UTC m=+1993.762571108" watchObservedRunningTime="2026-11-01 22:57:05.361865628 +0000 UTC m=+1993.766456259" Nov 01 22:57:05.905894 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:05.905853 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:05.913078 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:05.913046 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:06.344579 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:06.344547 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:06.345511 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:06.345491 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:57:09.003611 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:09.003575 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-788d7b-57874489c5-kkxpl"] Nov 01 22:57:09.009026 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:09.008995 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-788d7b-57874489c5-kkxpl" Nov 01 22:57:09.014816 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:09.014789 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-788d7b-dockercfg-s77jw\"" Nov 01 22:57:09.015375 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:09.015354 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-788d7b\"" Nov 01 22:57:09.018637 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:09.018597 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-788d7b-57874489c5-kkxpl"] Nov 01 22:57:09.161617 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:09.161573 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e080376-e7a9-406c-b102-cc3cd7c5ebcb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-788d7b-57874489c5-kkxpl\" (UID: \"7e080376-e7a9-406c-b102-cc3cd7c5ebcb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-788d7b-57874489c5-kkxpl" Nov 01 22:57:09.195858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:09.170246 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd85d\" (UniqueName: \"kubernetes.io/projected/7e080376-e7a9-406c-b102-cc3cd7c5ebcb-kube-api-access-kd85d\") pod \"payload-pre-processing-e2e-shared-a-w2-788d7b-57874489c5-kkxpl\" (UID: \"7e080376-e7a9-406c-b102-cc3cd7c5ebcb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-788d7b-57874489c5-kkxpl" Nov 01 22:57:09.411429 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:09.407373 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd85d\" (UniqueName: \"kubernetes.io/projected/7e080376-e7a9-406c-b102-cc3cd7c5ebcb-kube-api-access-kd85d\") pod \"payload-pre-processing-e2e-shared-a-w2-788d7b-57874489c5-kkxpl\" (UID: \"7e080376-e7a9-406c-b102-cc3cd7c5ebcb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-788d7b-57874489c5-kkxpl" Nov 01 22:57:09.411429 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:09.407718 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e080376-e7a9-406c-b102-cc3cd7c5ebcb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-788d7b-57874489c5-kkxpl\" (UID: \"7e080376-e7a9-406c-b102-cc3cd7c5ebcb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-788d7b-57874489c5-kkxpl" Nov 01 22:57:09.411429 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:09.408824 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e080376-e7a9-406c-b102-cc3cd7c5ebcb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-788d7b-57874489c5-kkxpl\" (UID: \"7e080376-e7a9-406c-b102-cc3cd7c5ebcb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-788d7b-57874489c5-kkxpl" Nov 01 22:57:09.422424 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:09.422391 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd85d\" (UniqueName: \"kubernetes.io/projected/7e080376-e7a9-406c-b102-cc3cd7c5ebcb-kube-api-access-kd85d\") pod \"payload-pre-processing-e2e-shared-a-w2-788d7b-57874489c5-kkxpl\" (UID: \"7e080376-e7a9-406c-b102-cc3cd7c5ebcb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-788d7b-57874489c5-kkxpl" Nov 01 22:57:09.631324 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:09.631291 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-788d7b-57874489c5-kkxpl" Nov 01 22:57:09.930921 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:09.930893 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-788d7b-57874489c5-kkxpl"] Nov 01 22:57:09.931333 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:57:09.931308 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e080376_e7a9_406c_b102_cc3cd7c5ebcb.slice/crio-b0f993cf54ea66c233b6e351c87fdc28ad25f74cb5884029a912671b919b67f1 WatchSource:0}: Error finding container b0f993cf54ea66c233b6e351c87fdc28ad25f74cb5884029a912671b919b67f1: Status 404 returned error can't find the container with id b0f993cf54ea66c233b6e351c87fdc28ad25f74cb5884029a912671b919b67f1 Nov 01 22:57:10.404152 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:10.404067 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-788d7b-57874489c5-kkxpl" event={"ID":"7e080376-e7a9-406c-b102-cc3cd7c5ebcb","Type":"ContainerStarted","Data":"526310d78ebbd50c674182729be237cedfd15e6c93b8942204a226d28c19d9db"} Nov 01 22:57:10.404152 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:10.404120 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-788d7b-57874489c5-kkxpl" event={"ID":"7e080376-e7a9-406c-b102-cc3cd7c5ebcb","Type":"ContainerStarted","Data":"b0f993cf54ea66c233b6e351c87fdc28ad25f74cb5884029a912671b919b67f1"} Nov 01 22:57:10.404639 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:10.404329 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-788d7b-57874489c5-kkxpl" Nov 01 22:57:10.426861 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:10.426789 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-788d7b-57874489c5-kkxpl" podStartSLOduration=2.426774311 podStartE2EDuration="2.426774311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:57:10.424116078 +0000 UTC m=+1998.828706721" watchObservedRunningTime="2026-11-01 22:57:10.426774311 +0000 UTC m=+1998.831364942" Nov 01 22:57:12.945976 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:12.945935 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:57:16.961416 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:57:16.961379 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:57:21.435527 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:21.435130 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-788d7b-57874489c5-kkxpl" Nov 01 22:57:27.803898 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:27.803858 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn"] Nov 01 22:57:27.809075 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:27.809047 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn" Nov 01 22:57:27.813910 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:27.813879 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-788d7b-serving-cert\"" Nov 01 22:57:27.821073 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:27.821040 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn"] Nov 01 22:57:27.947077 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:27.945697 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/129162f7-0fdd-426d-a66f-6acecdf14172-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn\" (UID: \"129162f7-0fdd-426d-a66f-6acecdf14172\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn" Nov 01 22:57:27.947077 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:27.945895 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz7gp\" (UniqueName: \"kubernetes.io/projected/129162f7-0fdd-426d-a66f-6acecdf14172-kube-api-access-vz7gp\") pod \"maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn\" (UID: \"129162f7-0fdd-426d-a66f-6acecdf14172\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn" Nov 01 22:57:27.947077 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:27.945931 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/129162f7-0fdd-426d-a66f-6acecdf14172-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn\" (UID: \"129162f7-0fdd-426d-a66f-6acecdf14172\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn" Nov 01 22:57:27.947077 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:27.945966 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/129162f7-0fdd-426d-a66f-6acecdf14172-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn\" (UID: \"129162f7-0fdd-426d-a66f-6acecdf14172\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn" Nov 01 22:57:28.058000 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:28.055752 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/129162f7-0fdd-426d-a66f-6acecdf14172-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn\" (UID: \"129162f7-0fdd-426d-a66f-6acecdf14172\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn" Nov 01 22:57:28.058000 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:28.055915 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz7gp\" (UniqueName: \"kubernetes.io/projected/129162f7-0fdd-426d-a66f-6acecdf14172-kube-api-access-vz7gp\") pod \"maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn\" (UID: \"129162f7-0fdd-426d-a66f-6acecdf14172\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn" Nov 01 22:57:28.058000 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:28.055948 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/129162f7-0fdd-426d-a66f-6acecdf14172-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn\" (UID: \"129162f7-0fdd-426d-a66f-6acecdf14172\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn" Nov 01 22:57:28.058000 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:28.055982 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/129162f7-0fdd-426d-a66f-6acecdf14172-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn\" (UID: \"129162f7-0fdd-426d-a66f-6acecdf14172\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn" Nov 01 22:57:28.058000 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:28.056706 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/129162f7-0fdd-426d-a66f-6acecdf14172-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn\" (UID: \"129162f7-0fdd-426d-a66f-6acecdf14172\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn" Nov 01 22:57:28.073017 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:28.072986 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz7gp\" (UniqueName: \"kubernetes.io/projected/129162f7-0fdd-426d-a66f-6acecdf14172-kube-api-access-vz7gp\") pod \"maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn\" (UID: \"129162f7-0fdd-426d-a66f-6acecdf14172\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn" Nov 01 22:57:28.079853 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:28.077995 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/129162f7-0fdd-426d-a66f-6acecdf14172-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn\" (UID: \"129162f7-0fdd-426d-a66f-6acecdf14172\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn" Nov 01 22:57:28.080917 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:28.080890 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/129162f7-0fdd-426d-a66f-6acecdf14172-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn\" (UID: \"129162f7-0fdd-426d-a66f-6acecdf14172\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn" Nov 01 22:57:28.148248 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:28.148208 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn" Nov 01 22:57:28.521320 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:28.518391 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn"] Nov 01 22:57:28.527582 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:57:28.526616 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod129162f7_0fdd_426d_a66f_6acecdf14172.slice/crio-19690b948edc57c3c02e24216d4ad151dc14735b96e625edd8bea2e49d142c02 WatchSource:0}: Error finding container 19690b948edc57c3c02e24216d4ad151dc14735b96e625edd8bea2e49d142c02: Status 404 returned error can't find the container with id 19690b948edc57c3c02e24216d4ad151dc14735b96e625edd8bea2e49d142c02 Nov 01 22:57:28.617904 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:28.617825 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn" event={"ID":"129162f7-0fdd-426d-a66f-6acecdf14172","Type":"ContainerStarted","Data":"19690b948edc57c3c02e24216d4ad151dc14735b96e625edd8bea2e49d142c02"} Nov 01 22:57:29.628301 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:29.628265 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn" event={"ID":"129162f7-0fdd-426d-a66f-6acecdf14172","Type":"ContainerStarted","Data":"e4a66c28b2d29ca75841b2c64a8b66873a0c56a8dcae8c67eff2bd14080e802e"} Nov 01 22:57:29.628782 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:29.628359 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn" Nov 01 22:57:29.651620 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:29.651561 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn" podStartSLOduration=2.232327082 podStartE2EDuration="2.651545761s" totalImagesPullingTime="419.218679ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:57:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:57:29.648931434 +0000 UTC m=+2018.053522067" watchObservedRunningTime="2026-11-01 22:57:29.651545761 +0000 UTC m=+2018.056136392" Nov 01 22:57:35.681808 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:35.681773 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn" Nov 01 22:57:46.962664 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:57:46.962620 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:57:54.056352 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.056316 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47"] Nov 01 22:57:54.064266 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.064224 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:57:54.069339 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.069307 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47"] Nov 01 22:57:54.119288 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.119251 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-d73b25-openshift-default-dockercfg-xl57x\"" Nov 01 22:57:54.159532 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.159489 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-istio-data\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:57:54.159758 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.159555 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-credential-socket\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:57:54.159758 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.159617 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-workload-certs\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:57:54.159758 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.159655 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-istio-envoy\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:57:54.159758 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.159682 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b301bf-5694-40b2-8789-07555de1ed0f-istiod-ca-cert\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:57:54.159976 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.159763 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-workload-socket\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:57:54.159976 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.159941 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h547g\" (UniqueName: \"kubernetes.io/projected/d7b301bf-5694-40b2-8789-07555de1ed0f-kube-api-access-h547g\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:57:54.160134 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.160117 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b301bf-5694-40b2-8789-07555de1ed0f-istio-podinfo\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:57:54.160177 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.160162 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7b301bf-5694-40b2-8789-07555de1ed0f-istio-token\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:57:54.281914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.266063 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b301bf-5694-40b2-8789-07555de1ed0f-istio-podinfo\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:57:54.281914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.266171 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7b301bf-5694-40b2-8789-07555de1ed0f-istio-token\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:57:54.281914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.266277 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-istio-data\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:57:54.281914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.266350 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-credential-socket\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:57:54.281914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.266379 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-workload-certs\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:57:54.281914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.266441 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-istio-envoy\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:57:54.281914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.266485 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b301bf-5694-40b2-8789-07555de1ed0f-istiod-ca-cert\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:57:54.281914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.266513 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-workload-socket\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:57:54.281914 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.266647 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h547g\" (UniqueName: \"kubernetes.io/projected/d7b301bf-5694-40b2-8789-07555de1ed0f-kube-api-access-h547g\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:57:54.289871 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.289117 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-workload-certs\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:57:54.289871 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.289700 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-istio-data\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:57:54.290135 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.289975 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-credential-socket\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:57:54.290704 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.290675 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b301bf-5694-40b2-8789-07555de1ed0f-istiod-ca-cert\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:57:54.291197 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.291167 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-workload-socket\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:57:54.326933 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.303115 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b301bf-5694-40b2-8789-07555de1ed0f-istio-podinfo\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:57:54.340866 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.336575 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7b301bf-5694-40b2-8789-07555de1ed0f-istio-token\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:57:54.348857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.341631 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h547g\" (UniqueName: \"kubernetes.io/projected/d7b301bf-5694-40b2-8789-07555de1ed0f-kube-api-access-h547g\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:57:54.348857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.346006 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-istio-envoy\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:57:54.400907 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.396133 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:57:54.679248 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.678278 5011 scope.go:122] "RemoveContainer" containerID="1c253276b4586b6b4d42b5633d9cf55921ce6389b9864df8025860fef89b837f" Nov 01 22:57:54.858685 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.858644 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47"] Nov 01 22:57:54.859228 ip-10-0-142-238 kubenswrapper[5011]: W1101 22:57:54.859199 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7b301bf_5694_40b2_8789_07555de1ed0f.slice/crio-ad1b910f015be24b01f78d0adf542821c09005b7ca4308df0470952b83b08e2f WatchSource:0}: Error finding container ad1b910f015be24b01f78d0adf542821c09005b7ca4308df0470952b83b08e2f: Status 404 returned error can't find the container with id ad1b910f015be24b01f78d0adf542821c09005b7ca4308df0470952b83b08e2f Nov 01 22:57:54.862316 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.862275 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:57:54.862433 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.862351 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:57:54.862433 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.862381 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:57:54.939129 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:54.939099 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" event={"ID":"d7b301bf-5694-40b2-8789-07555de1ed0f","Type":"ContainerStarted","Data":"ad1b910f015be24b01f78d0adf542821c09005b7ca4308df0470952b83b08e2f"} Nov 01 22:57:55.985859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:55.984651 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" event={"ID":"d7b301bf-5694-40b2-8789-07555de1ed0f","Type":"ContainerStarted","Data":"10375264c2c86cd17cb06109c5d4e5876ab93023af79de115915f4008c874104"} Nov 01 22:57:56.014492 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:56.014425 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" podStartSLOduration=2.014408154 podStartE2EDuration="2.014408154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:57:56.012578382 +0000 UTC m=+2044.417169013" watchObservedRunningTime="2026-11-01 22:57:56.014408154 +0000 UTC m=+2044.418998787" Nov 01 22:57:56.397670 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:56.397576 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:57:56.403985 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:56.403958 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:57:56.992714 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:56.992682 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:57:56.994060 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:57:56.994035 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:58:00.020168 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:00.020129 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d73b25-7ffc64f5d5-7wcv8"] Nov 01 22:58:00.042070 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:00.042037 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d73b25-7ffc64f5d5-7wcv8" Nov 01 22:58:00.045853 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:00.045804 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d73b25-7ffc64f5d5-7wcv8"] Nov 01 22:58:00.105578 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:00.105544 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-d73b25\"" Nov 01 22:58:00.105991 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:00.105969 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-d73b25-dockercfg-hm5m7\"" Nov 01 22:58:00.151580 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:00.137408 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79ppt\" (UniqueName: \"kubernetes.io/projected/8fee5226-37e2-4c83-9225-449df015af93-kube-api-access-79ppt\") pod \"payload-processing-e2e-shared-b-w2-d73b25-7ffc64f5d5-7wcv8\" (UID: \"8fee5226-37e2-4c83-9225-449df015af93\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d73b25-7ffc64f5d5-7wcv8" Nov 01 22:58:00.164856 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:00.152200 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fee5226-37e2-4c83-9225-449df015af93-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d73b25-7ffc64f5d5-7wcv8\" (UID: \"8fee5226-37e2-4c83-9225-449df015af93\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d73b25-7ffc64f5d5-7wcv8" Nov 01 22:58:00.320913 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:00.276721 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79ppt\" (UniqueName: \"kubernetes.io/projected/8fee5226-37e2-4c83-9225-449df015af93-kube-api-access-79ppt\") pod \"payload-processing-e2e-shared-b-w2-d73b25-7ffc64f5d5-7wcv8\" (UID: \"8fee5226-37e2-4c83-9225-449df015af93\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d73b25-7ffc64f5d5-7wcv8" Nov 01 22:58:00.320913 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:00.276965 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fee5226-37e2-4c83-9225-449df015af93-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d73b25-7ffc64f5d5-7wcv8\" (UID: \"8fee5226-37e2-4c83-9225-449df015af93\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d73b25-7ffc64f5d5-7wcv8" Nov 01 22:58:00.320913 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:00.278055 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fee5226-37e2-4c83-9225-449df015af93-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d73b25-7ffc64f5d5-7wcv8\" (UID: \"8fee5226-37e2-4c83-9225-449df015af93\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d73b25-7ffc64f5d5-7wcv8" Nov 01 22:58:00.415676 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:00.414697 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79ppt\" (UniqueName: \"kubernetes.io/projected/8fee5226-37e2-4c83-9225-449df015af93-kube-api-access-79ppt\") pod \"payload-processing-e2e-shared-b-w2-d73b25-7ffc64f5d5-7wcv8\" (UID: \"8fee5226-37e2-4c83-9225-449df015af93\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d73b25-7ffc64f5d5-7wcv8" Nov 01 22:58:00.475381 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:00.475346 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d73b25-7ffc64f5d5-7wcv8" Nov 01 22:58:00.826658 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:00.826608 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d73b25-7ffc64f5d5-7wcv8"] Nov 01 22:58:01.035578 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:01.035538 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d73b25-7ffc64f5d5-7wcv8" event={"ID":"8fee5226-37e2-4c83-9225-449df015af93","Type":"ContainerStarted","Data":"1f8067b1b20d768b1a845887a55a74547510e04d9a8b8de060b307782ea48728"} Nov 01 22:58:01.036042 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:01.035640 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d73b25-7ffc64f5d5-7wcv8" event={"ID":"8fee5226-37e2-4c83-9225-449df015af93","Type":"ContainerStarted","Data":"a14294edfffc41182f8ee6740a4ef7c32f8bd806fe45d7f72e94f5c8af183c4e"} Nov 01 22:58:01.036042 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:01.035856 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d73b25-7ffc64f5d5-7wcv8" Nov 01 22:58:01.053769 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:01.053714 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d73b25-7ffc64f5d5-7wcv8" podStartSLOduration=2.0537007369999998 podStartE2EDuration="2.053700737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:58:01.052346342 +0000 UTC m=+2049.456936975" watchObservedRunningTime="2026-11-01 22:58:01.053700737 +0000 UTC m=+2049.458291369" Nov 01 22:58:11.023579 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:11.023549 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 22:58:12.082612 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:12.082571 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d73b25-7ffc64f5d5-7wcv8" Nov 01 22:58:16.981295 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:58:16.981252 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:58:22.386023 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:22.385982 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:58:22.386541 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:22.386054 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" podUID="a03efd24-da7e-407d-896d-d22e46206274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:58:25.459713 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:25.459676 5011 generic.go:356] "Generic (PLEG): container finished" podID="a03efd24-da7e-407d-896d-d22e46206274" containerID="53a9b720e37ce41acd3fb311faa168f34d9b506fa7322cbe682798a67b036677" exitCode=0 Nov 01 22:58:25.460202 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:25.459752 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" event={"ID":"a03efd24-da7e-407d-896d-d22e46206274","Type":"ContainerDied","Data":"53a9b720e37ce41acd3fb311faa168f34d9b506fa7322cbe682798a67b036677"} Nov 01 22:58:25.460277 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:25.460263 5011 scope.go:122] "RemoveContainer" containerID="53a9b720e37ce41acd3fb311faa168f34d9b506fa7322cbe682798a67b036677" Nov 01 22:58:25.462280 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:25.462251 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:58:25.462352 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:25.462321 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:58:25.462391 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:25.462354 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:58:26.486857 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:26.483524 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" event={"ID":"a03efd24-da7e-407d-896d-d22e46206274","Type":"ContainerStarted","Data":"96732250585ad7144915c56e1032f185f361916b09c50c87095fdddf59906e75"} Nov 01 22:58:26.906450 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:26.906353 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:58:26.906646 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:26.906622 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Nov 01 22:58:26.906709 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:26.906689 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" podUID="a03efd24-da7e-407d-896d-d22e46206274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Nov 01 22:58:27.906721 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:27.906684 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Nov 01 22:58:27.907145 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:27.906754 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" podUID="a03efd24-da7e-407d-896d-d22e46206274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Nov 01 22:58:28.635418 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:28.626462 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr"] Nov 01 22:58:28.635418 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:28.626721 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr" podUID="39d06520-7ba9-44c0-a35d-22cfce8aaf08" containerName="maas-api" containerID="cri-o://b8ac5cc4ebe6f774ea9658e32cc36197dd8fbf9b7883b18f83a516c611a2afb9" gracePeriod=30 Nov 01 22:58:28.907913 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:28.907252 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Nov 01 22:58:28.907913 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:28.907334 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" podUID="a03efd24-da7e-407d-896d-d22e46206274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Nov 01 22:58:29.529704 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:29.528404 5011 generic.go:356] "Generic (PLEG): container finished" podID="39d06520-7ba9-44c0-a35d-22cfce8aaf08" containerID="b8ac5cc4ebe6f774ea9658e32cc36197dd8fbf9b7883b18f83a516c611a2afb9" exitCode=0 Nov 01 22:58:29.529704 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:29.528483 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr" event={"ID":"39d06520-7ba9-44c0-a35d-22cfce8aaf08","Type":"ContainerDied","Data":"b8ac5cc4ebe6f774ea9658e32cc36197dd8fbf9b7883b18f83a516c611a2afb9"} Nov 01 22:58:29.529704 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:29.528549 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr" event={"ID":"39d06520-7ba9-44c0-a35d-22cfce8aaf08","Type":"ContainerDied","Data":"e094192df98e175aa9ee2922163fdd8b1a04ceaee6bef9ae9362956d25fab98d"} Nov 01 22:58:29.529704 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:29.528572 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e094192df98e175aa9ee2922163fdd8b1a04ceaee6bef9ae9362956d25fab98d" Nov 01 22:58:29.731303 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:29.731270 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr" Nov 01 22:58:29.892312 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:29.892274 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39d06520-7ba9-44c0-a35d-22cfce8aaf08-metrics-certs\" (UniqueName: \"kubernetes.io/secret/39d06520-7ba9-44c0-a35d-22cfce8aaf08-metrics-certs\") pod \"39d06520-7ba9-44c0-a35d-22cfce8aaf08\" (UID: \"39d06520-7ba9-44c0-a35d-22cfce8aaf08\") " Nov 01 22:58:29.892618 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:29.892585 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39d06520-7ba9-44c0-a35d-22cfce8aaf08-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39d06520-7ba9-44c0-a35d-22cfce8aaf08-maas-api-tls\") pod \"39d06520-7ba9-44c0-a35d-22cfce8aaf08\" (UID: \"39d06520-7ba9-44c0-a35d-22cfce8aaf08\") " Nov 01 22:58:29.892926 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:29.892907 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39d06520-7ba9-44c0-a35d-22cfce8aaf08-kube-api-access-542wn\" (UniqueName: \"kubernetes.io/projected/39d06520-7ba9-44c0-a35d-22cfce8aaf08-kube-api-access-542wn\") pod \"39d06520-7ba9-44c0-a35d-22cfce8aaf08\" (UID: \"39d06520-7ba9-44c0-a35d-22cfce8aaf08\") " Nov 01 22:58:29.893234 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:29.893210 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39d06520-7ba9-44c0-a35d-22cfce8aaf08-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39d06520-7ba9-44c0-a35d-22cfce8aaf08-openshift-service-ca\") pod \"39d06520-7ba9-44c0-a35d-22cfce8aaf08\" (UID: \"39d06520-7ba9-44c0-a35d-22cfce8aaf08\") " Nov 01 22:58:29.895728 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:29.895691 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39d06520-7ba9-44c0-a35d-22cfce8aaf08-openshift-service-ca" pod "39d06520-7ba9-44c0-a35d-22cfce8aaf08" (UID: "39d06520-7ba9-44c0-a35d-22cfce8aaf08"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:58:29.899113 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:29.899082 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39d06520-7ba9-44c0-a35d-22cfce8aaf08-metrics-certs" pod "39d06520-7ba9-44c0-a35d-22cfce8aaf08" (UID: "39d06520-7ba9-44c0-a35d-22cfce8aaf08"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:58:29.904239 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:29.904159 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39d06520-7ba9-44c0-a35d-22cfce8aaf08-kube-api-access-542wn" pod "39d06520-7ba9-44c0-a35d-22cfce8aaf08" (UID: "39d06520-7ba9-44c0-a35d-22cfce8aaf08"). InnerVolumeSpecName "kube-api-access-542wn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:58:29.912137 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:29.912098 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39d06520-7ba9-44c0-a35d-22cfce8aaf08-maas-api-tls" pod "39d06520-7ba9-44c0-a35d-22cfce8aaf08" (UID: "39d06520-7ba9-44c0-a35d-22cfce8aaf08"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:58:29.912548 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:29.912511 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Nov 01 22:58:29.912627 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:29.912583 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" podUID="a03efd24-da7e-407d-896d-d22e46206274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Nov 01 22:58:29.996917 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:29.996279 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-542wn\" (UniqueName: \"kubernetes.io/projected/39d06520-7ba9-44c0-a35d-22cfce8aaf08-kube-api-access-542wn\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:29.996917 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:29.996316 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39d06520-7ba9-44c0-a35d-22cfce8aaf08-openshift-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:29.996917 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:29.996334 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39d06520-7ba9-44c0-a35d-22cfce8aaf08-metrics-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:29.996917 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:29.996348 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39d06520-7ba9-44c0-a35d-22cfce8aaf08-maas-api-tls\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:30.578858 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:30.578487 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr" Nov 01 22:58:30.672683 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:30.672644 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr"] Nov 01 22:58:30.679555 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:30.679522 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6b78bd-55bf894577-tv2wr"] Nov 01 22:58:30.906392 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:30.906306 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Nov 01 22:58:30.906392 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:30.906370 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" podUID="a03efd24-da7e-407d-896d-d22e46206274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Nov 01 22:58:31.772937 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:31.772900 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39d06520-7ba9-44c0-a35d-22cfce8aaf08" path="/var/lib/kubelet/pods/39d06520-7ba9-44c0-a35d-22cfce8aaf08/volumes" Nov 01 22:58:31.906074 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:31.906039 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Nov 01 22:58:31.906263 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:31.906095 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" podUID="a03efd24-da7e-407d-896d-d22e46206274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Nov 01 22:58:32.911183 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:32.911153 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:58:32.911709 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:32.911687 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:58:32.912551 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:32.912531 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 22:58:36.124205 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:36.124165 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr"] Nov 01 22:58:36.124701 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:36.124436 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" podUID="ee08f7f3-27a6-44fc-9099-dbe8459e6b12" containerName="istio-proxy" containerID="cri-o://f16d573efb1b8b90ba0416907e229aed0beb62e88fa7f0546b2f83b48ed7e0ed" gracePeriod=30 Nov 01 22:58:36.130072 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:36.130018 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns"] Nov 01 22:58:36.130496 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:36.130452 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" podUID="00381713-efff-4f16-8fc8-af879b07e517" containerName="istio-proxy" containerID="cri-o://466d89db26928b860cbaf6d4ac0ae763c7f02f4bac4a1318cf781dd6ed487ff5" gracePeriod=30 Nov 01 22:58:36.134901 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:36.134875 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp"] Nov 01 22:58:36.135355 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:36.135325 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" podUID="831636af-90aa-41a1-b03f-561c0462ab4e" containerName="istio-proxy" containerID="cri-o://ce03038a96f765ded4e8c635d32dab68dd04dd499bcd3402698c9588b828b4bc" gracePeriod=30 Nov 01 22:58:39.277903 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:39.277852 5011 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:58:39.278299 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:39.277922 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" podUID="831636af-90aa-41a1-b03f-561c0462ab4e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:58:40.127185 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:40.127150 5011 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:58:40.127363 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:40.127211 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" podUID="ee08f7f3-27a6-44fc-9099-dbe8459e6b12" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:58:40.436281 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:40.436239 5011 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:58:40.436747 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:40.436321 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" podUID="00381713-efff-4f16-8fc8-af879b07e517" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:58:41.462887 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.462807 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:58:41.556919 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.556893 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:58:41.560954 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.560933 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:58:41.639192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.634002 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-workload-certs\") pod \"831636af-90aa-41a1-b03f-561c0462ab4e\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " Nov 01 22:58:41.639192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.634082 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00381713-efff-4f16-8fc8-af879b07e517-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00381713-efff-4f16-8fc8-af879b07e517-istio-podinfo\") pod \"00381713-efff-4f16-8fc8-af879b07e517\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " Nov 01 22:58:41.639192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.634123 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00381713-efff-4f16-8fc8-af879b07e517-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00381713-efff-4f16-8fc8-af879b07e517-istiod-ca-cert\") pod \"00381713-efff-4f16-8fc8-af879b07e517\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " Nov 01 22:58:41.639192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.634156 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istio-podinfo\") pod \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " Nov 01 22:58:41.639192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.634321 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-istio-data\") pod \"00381713-efff-4f16-8fc8-af879b07e517\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " Nov 01 22:58:41.639192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.634391 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-kube-api-access-xtggj\" (UniqueName: \"kubernetes.io/projected/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-kube-api-access-xtggj\") pod \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " Nov 01 22:58:41.639192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.634444 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-workload-certs\") pod \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " Nov 01 22:58:41.639192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.634490 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/831636af-90aa-41a1-b03f-561c0462ab4e-istio-token\" (UniqueName: \"kubernetes.io/projected/831636af-90aa-41a1-b03f-561c0462ab4e-istio-token\") pod \"831636af-90aa-41a1-b03f-561c0462ab4e\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " Nov 01 22:58:41.639192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.634520 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istio-token\" (UniqueName: \"kubernetes.io/projected/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istio-token\") pod \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " Nov 01 22:58:41.639192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.634555 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istiod-ca-cert\") pod \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " Nov 01 22:58:41.639192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.634596 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/831636af-90aa-41a1-b03f-561c0462ab4e-kube-api-access-v7cwd\" (UniqueName: \"kubernetes.io/projected/831636af-90aa-41a1-b03f-561c0462ab4e-kube-api-access-v7cwd\") pod \"831636af-90aa-41a1-b03f-561c0462ab4e\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " Nov 01 22:58:41.639192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.634648 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istio-data\") pod \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " Nov 01 22:58:41.639192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.634693 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-istio-envoy\") pod \"00381713-efff-4f16-8fc8-af879b07e517\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " Nov 01 22:58:41.639192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.634737 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-credential-socket\") pod \"831636af-90aa-41a1-b03f-561c0462ab4e\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " Nov 01 22:58:41.639192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.634768 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istio-envoy\") pod \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " Nov 01 22:58:41.639192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.634799 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-credential-socket\") pod \"00381713-efff-4f16-8fc8-af879b07e517\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " Nov 01 22:58:41.639192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.634850 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-workload-socket\") pod \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " Nov 01 22:58:41.639192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.634907 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00381713-efff-4f16-8fc8-af879b07e517-kube-api-access-9vl26\" (UniqueName: \"kubernetes.io/projected/00381713-efff-4f16-8fc8-af879b07e517-kube-api-access-9vl26\") pod \"00381713-efff-4f16-8fc8-af879b07e517\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " Nov 01 22:58:41.639192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.634944 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/831636af-90aa-41a1-b03f-561c0462ab4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/831636af-90aa-41a1-b03f-561c0462ab4e-istio-podinfo\") pod \"831636af-90aa-41a1-b03f-561c0462ab4e\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " Nov 01 22:58:41.639192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.635051 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-workload-socket\") pod \"831636af-90aa-41a1-b03f-561c0462ab4e\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " Nov 01 22:58:41.639192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.635103 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-credential-socket\") pod \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\" (UID: \"ee08f7f3-27a6-44fc-9099-dbe8459e6b12\") " Nov 01 22:58:41.639192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.635338 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00381713-efff-4f16-8fc8-af879b07e517-istio-token\" (UniqueName: \"kubernetes.io/projected/00381713-efff-4f16-8fc8-af879b07e517-istio-token\") pod \"00381713-efff-4f16-8fc8-af879b07e517\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " Nov 01 22:58:41.639192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.635383 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-istio-data\") pod \"831636af-90aa-41a1-b03f-561c0462ab4e\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " Nov 01 22:58:41.639192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.635500 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-workload-socket\") pod \"00381713-efff-4f16-8fc8-af879b07e517\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " Nov 01 22:58:41.639192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.635560 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-workload-certs\") pod \"00381713-efff-4f16-8fc8-af879b07e517\" (UID: \"00381713-efff-4f16-8fc8-af879b07e517\") " Nov 01 22:58:41.639192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.635599 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-istio-envoy\") pod \"831636af-90aa-41a1-b03f-561c0462ab4e\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " Nov 01 22:58:41.639192 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.635682 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/831636af-90aa-41a1-b03f-561c0462ab4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/831636af-90aa-41a1-b03f-561c0462ab4e-istiod-ca-cert\") pod \"831636af-90aa-41a1-b03f-561c0462ab4e\" (UID: \"831636af-90aa-41a1-b03f-561c0462ab4e\") " Nov 01 22:58:41.663403 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.641734 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istiod-ca-cert" pod "ee08f7f3-27a6-44fc-9099-dbe8459e6b12" (UID: "ee08f7f3-27a6-44fc-9099-dbe8459e6b12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:58:41.663403 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.643482 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-workload-certs" pod "831636af-90aa-41a1-b03f-561c0462ab4e" (UID: "831636af-90aa-41a1-b03f-561c0462ab4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.663403 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.643910 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-workload-certs" pod "ee08f7f3-27a6-44fc-9099-dbe8459e6b12" (UID: "ee08f7f3-27a6-44fc-9099-dbe8459e6b12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.663403 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.644225 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-credential-socket" pod "ee08f7f3-27a6-44fc-9099-dbe8459e6b12" (UID: "ee08f7f3-27a6-44fc-9099-dbe8459e6b12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.663403 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.644576 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-credential-socket" pod "00381713-efff-4f16-8fc8-af879b07e517" (UID: "00381713-efff-4f16-8fc8-af879b07e517"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.663403 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.644813 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-workload-socket" pod "ee08f7f3-27a6-44fc-9099-dbe8459e6b12" (UID: "ee08f7f3-27a6-44fc-9099-dbe8459e6b12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.663403 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.650245 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00381713-efff-4f16-8fc8-af879b07e517-istiod-ca-cert" pod "00381713-efff-4f16-8fc8-af879b07e517" (UID: "00381713-efff-4f16-8fc8-af879b07e517"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:58:41.663403 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.650542 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-istio-data" pod "00381713-efff-4f16-8fc8-af879b07e517" (UID: "00381713-efff-4f16-8fc8-af879b07e517"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.663403 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.660081 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-workload-socket" pod "00381713-efff-4f16-8fc8-af879b07e517" (UID: "00381713-efff-4f16-8fc8-af879b07e517"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.681630 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.667143 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-istio-data" pod "831636af-90aa-41a1-b03f-561c0462ab4e" (UID: "831636af-90aa-41a1-b03f-561c0462ab4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.681630 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.667241 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-credential-socket" pod "831636af-90aa-41a1-b03f-561c0462ab4e" (UID: "831636af-90aa-41a1-b03f-561c0462ab4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.681630 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.675102 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/831636af-90aa-41a1-b03f-561c0462ab4e-istiod-ca-cert" pod "831636af-90aa-41a1-b03f-561c0462ab4e" (UID: "831636af-90aa-41a1-b03f-561c0462ab4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:58:41.681630 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.678378 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-workload-socket" pod "831636af-90aa-41a1-b03f-561c0462ab4e" (UID: "831636af-90aa-41a1-b03f-561c0462ab4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.681630 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.679586 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-workload-certs" pod "00381713-efff-4f16-8fc8-af879b07e517" (UID: "00381713-efff-4f16-8fc8-af879b07e517"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.683346 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.682756 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istio-data" pod "ee08f7f3-27a6-44fc-9099-dbe8459e6b12" (UID: "ee08f7f3-27a6-44fc-9099-dbe8459e6b12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.688233 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.688181 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-kube-api-access-xtggj" pod "ee08f7f3-27a6-44fc-9099-dbe8459e6b12" (UID: "ee08f7f3-27a6-44fc-9099-dbe8459e6b12"). InnerVolumeSpecName "kube-api-access-xtggj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:58:41.718347 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.718306 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/831636af-90aa-41a1-b03f-561c0462ab4e-istio-token" pod "831636af-90aa-41a1-b03f-561c0462ab4e" (UID: "831636af-90aa-41a1-b03f-561c0462ab4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:58:41.718676 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.718460 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/831636af-90aa-41a1-b03f-561c0462ab4e-istio-podinfo" pod "831636af-90aa-41a1-b03f-561c0462ab4e" (UID: "831636af-90aa-41a1-b03f-561c0462ab4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:58:41.718676 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.718572 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00381713-efff-4f16-8fc8-af879b07e517-istio-podinfo" pod "00381713-efff-4f16-8fc8-af879b07e517" (UID: "00381713-efff-4f16-8fc8-af879b07e517"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:58:41.718815 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.718688 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istio-podinfo" pod "ee08f7f3-27a6-44fc-9099-dbe8459e6b12" (UID: "ee08f7f3-27a6-44fc-9099-dbe8459e6b12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:58:41.718926 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.718882 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00381713-efff-4f16-8fc8-af879b07e517-istio-token" pod "00381713-efff-4f16-8fc8-af879b07e517" (UID: "00381713-efff-4f16-8fc8-af879b07e517"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:58:41.718987 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.718960 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00381713-efff-4f16-8fc8-af879b07e517-kube-api-access-9vl26" pod "00381713-efff-4f16-8fc8-af879b07e517" (UID: "00381713-efff-4f16-8fc8-af879b07e517"). InnerVolumeSpecName "kube-api-access-9vl26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:58:41.719151 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.719125 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istio-envoy" pod "ee08f7f3-27a6-44fc-9099-dbe8459e6b12" (UID: "ee08f7f3-27a6-44fc-9099-dbe8459e6b12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.719220 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.719205 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-istio-envoy" pod "00381713-efff-4f16-8fc8-af879b07e517" (UID: "00381713-efff-4f16-8fc8-af879b07e517"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.719352 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.719336 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/831636af-90aa-41a1-b03f-561c0462ab4e-kube-api-access-v7cwd" pod "831636af-90aa-41a1-b03f-561c0462ab4e" (UID: "831636af-90aa-41a1-b03f-561c0462ab4e"). InnerVolumeSpecName "kube-api-access-v7cwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:58:41.721958 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.721282 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istio-token" pod "ee08f7f3-27a6-44fc-9099-dbe8459e6b12" (UID: "ee08f7f3-27a6-44fc-9099-dbe8459e6b12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:58:41.725314 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.724951 5011 generic.go:356] "Generic (PLEG): container finished" podID="ee08f7f3-27a6-44fc-9099-dbe8459e6b12" containerID="f16d573efb1b8b90ba0416907e229aed0beb62e88fa7f0546b2f83b48ed7e0ed" exitCode=0 Nov 01 22:58:41.725314 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.725046 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" Nov 01 22:58:41.725314 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.725171 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" event={"ID":"ee08f7f3-27a6-44fc-9099-dbe8459e6b12","Type":"ContainerDied","Data":"f16d573efb1b8b90ba0416907e229aed0beb62e88fa7f0546b2f83b48ed7e0ed"} Nov 01 22:58:41.725314 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.725206 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr" event={"ID":"ee08f7f3-27a6-44fc-9099-dbe8459e6b12","Type":"ContainerDied","Data":"9f642835dfb492dae66015066b3d1e008807919cd26b73511d145da33f2c8a95"} Nov 01 22:58:41.725314 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.725226 5011 scope.go:122] "RemoveContainer" containerID="f16d573efb1b8b90ba0416907e229aed0beb62e88fa7f0546b2f83b48ed7e0ed" Nov 01 22:58:41.728073 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.728033 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-istio-envoy" pod "831636af-90aa-41a1-b03f-561c0462ab4e" (UID: "831636af-90aa-41a1-b03f-561c0462ab4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.729961 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.729815 5011 generic.go:356] "Generic (PLEG): container finished" podID="00381713-efff-4f16-8fc8-af879b07e517" containerID="466d89db26928b860cbaf6d4ac0ae763c7f02f4bac4a1318cf781dd6ed487ff5" exitCode=0 Nov 01 22:58:41.730195 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.730095 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" event={"ID":"00381713-efff-4f16-8fc8-af879b07e517","Type":"ContainerDied","Data":"466d89db26928b860cbaf6d4ac0ae763c7f02f4bac4a1318cf781dd6ed487ff5"} Nov 01 22:58:41.730195 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.730149 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" event={"ID":"00381713-efff-4f16-8fc8-af879b07e517","Type":"ContainerDied","Data":"1ba7fca4cf6bbba41b5676b7624d04b8a0a01a310b0161d002fe8229db9c34e5"} Nov 01 22:58:41.730195 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.730172 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns" Nov 01 22:58:41.731818 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.731794 5011 generic.go:356] "Generic (PLEG): container finished" podID="831636af-90aa-41a1-b03f-561c0462ab4e" containerID="ce03038a96f765ded4e8c635d32dab68dd04dd499bcd3402698c9588b828b4bc" exitCode=0 Nov 01 22:58:41.731939 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.731876 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" event={"ID":"831636af-90aa-41a1-b03f-561c0462ab4e","Type":"ContainerDied","Data":"ce03038a96f765ded4e8c635d32dab68dd04dd499bcd3402698c9588b828b4bc"} Nov 01 22:58:41.731939 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.731916 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" event={"ID":"831636af-90aa-41a1-b03f-561c0462ab4e","Type":"ContainerDied","Data":"1f980635acef8d1917619003d75a21524a5c8836518cf06cc14d15b857574d89"} Nov 01 22:58:41.732065 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.731984 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp" Nov 01 22:58:41.738200 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.738175 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vl26\" (UniqueName: \"kubernetes.io/projected/00381713-efff-4f16-8fc8-af879b07e517-kube-api-access-9vl26\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.738306 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.738206 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/831636af-90aa-41a1-b03f-561c0462ab4e-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.738306 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.738222 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.738306 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.738234 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.738306 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.738249 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00381713-efff-4f16-8fc8-af879b07e517-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.738306 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.738262 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.738306 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.738275 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.738306 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.738287 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.738306 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.738300 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.738746 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.738313 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/831636af-90aa-41a1-b03f-561c0462ab4e-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.738746 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.738328 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.738746 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.738341 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00381713-efff-4f16-8fc8-af879b07e517-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.738746 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.738355 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00381713-efff-4f16-8fc8-af879b07e517-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.738746 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.738371 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.738746 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.738384 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.738746 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.738398 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtggj\" (UniqueName: \"kubernetes.io/projected/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-kube-api-access-xtggj\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.738746 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.738412 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.738746 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.738425 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/831636af-90aa-41a1-b03f-561c0462ab4e-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.738746 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.738440 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.738746 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.738453 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.738746 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.738468 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7cwd\" (UniqueName: \"kubernetes.io/projected/831636af-90aa-41a1-b03f-561c0462ab4e-kube-api-access-v7cwd\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.738746 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.738484 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.738746 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.738496 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.738746 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.738510 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/831636af-90aa-41a1-b03f-561c0462ab4e-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.738746 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.738524 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.738746 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.738537 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00381713-efff-4f16-8fc8-af879b07e517-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.738746 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.738551 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee08f7f3-27a6-44fc-9099-dbe8459e6b12-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.754890 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.754864 5011 scope.go:122] "RemoveContainer" containerID="f16d573efb1b8b90ba0416907e229aed0beb62e88fa7f0546b2f83b48ed7e0ed" Nov 01 22:58:41.755356 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:58:41.755318 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f16d573efb1b8b90ba0416907e229aed0beb62e88fa7f0546b2f83b48ed7e0ed\": container with ID starting with f16d573efb1b8b90ba0416907e229aed0beb62e88fa7f0546b2f83b48ed7e0ed not found: ID does not exist" containerID="f16d573efb1b8b90ba0416907e229aed0beb62e88fa7f0546b2f83b48ed7e0ed" Nov 01 22:58:41.755473 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.755366 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f16d573efb1b8b90ba0416907e229aed0beb62e88fa7f0546b2f83b48ed7e0ed"} err="failed to get container status \"f16d573efb1b8b90ba0416907e229aed0beb62e88fa7f0546b2f83b48ed7e0ed\": rpc error: code = NotFound desc = could not find container \"f16d573efb1b8b90ba0416907e229aed0beb62e88fa7f0546b2f83b48ed7e0ed\": container with ID starting with f16d573efb1b8b90ba0416907e229aed0beb62e88fa7f0546b2f83b48ed7e0ed not found: ID does not exist" Nov 01 22:58:41.755473 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.755413 5011 scope.go:122] "RemoveContainer" containerID="466d89db26928b860cbaf6d4ac0ae763c7f02f4bac4a1318cf781dd6ed487ff5" Nov 01 22:58:41.770121 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.770098 5011 scope.go:122] "RemoveContainer" containerID="466d89db26928b860cbaf6d4ac0ae763c7f02f4bac4a1318cf781dd6ed487ff5" Nov 01 22:58:41.770436 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:58:41.770414 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"466d89db26928b860cbaf6d4ac0ae763c7f02f4bac4a1318cf781dd6ed487ff5\": container with ID starting with 466d89db26928b860cbaf6d4ac0ae763c7f02f4bac4a1318cf781dd6ed487ff5 not found: ID does not exist" containerID="466d89db26928b860cbaf6d4ac0ae763c7f02f4bac4a1318cf781dd6ed487ff5" Nov 01 22:58:41.770509 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.770446 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"466d89db26928b860cbaf6d4ac0ae763c7f02f4bac4a1318cf781dd6ed487ff5"} err="failed to get container status \"466d89db26928b860cbaf6d4ac0ae763c7f02f4bac4a1318cf781dd6ed487ff5\": rpc error: code = NotFound desc = could not find container \"466d89db26928b860cbaf6d4ac0ae763c7f02f4bac4a1318cf781dd6ed487ff5\": container with ID starting with 466d89db26928b860cbaf6d4ac0ae763c7f02f4bac4a1318cf781dd6ed487ff5 not found: ID does not exist" Nov 01 22:58:41.770509 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.770464 5011 scope.go:122] "RemoveContainer" containerID="ce03038a96f765ded4e8c635d32dab68dd04dd499bcd3402698c9588b828b4bc" Nov 01 22:58:41.772137 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.772088 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns"] Nov 01 22:58:41.778049 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.778029 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-wwmns"] Nov 01 22:58:41.784683 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.784663 5011 scope.go:122] "RemoveContainer" containerID="ce03038a96f765ded4e8c635d32dab68dd04dd499bcd3402698c9588b828b4bc" Nov 01 22:58:41.785257 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:58:41.785227 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce03038a96f765ded4e8c635d32dab68dd04dd499bcd3402698c9588b828b4bc\": container with ID starting with ce03038a96f765ded4e8c635d32dab68dd04dd499bcd3402698c9588b828b4bc not found: ID does not exist" containerID="ce03038a96f765ded4e8c635d32dab68dd04dd499bcd3402698c9588b828b4bc" Nov 01 22:58:41.785365 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.785268 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce03038a96f765ded4e8c635d32dab68dd04dd499bcd3402698c9588b828b4bc"} err="failed to get container status \"ce03038a96f765ded4e8c635d32dab68dd04dd499bcd3402698c9588b828b4bc\": rpc error: code = NotFound desc = could not find container \"ce03038a96f765ded4e8c635d32dab68dd04dd499bcd3402698c9588b828b4bc\": container with ID starting with ce03038a96f765ded4e8c635d32dab68dd04dd499bcd3402698c9588b828b4bc not found: ID does not exist" Nov 01 22:58:41.792093 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.792060 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr"] Nov 01 22:58:41.796478 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.796450 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jvjxr"] Nov 01 22:58:41.808918 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.808880 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp"] Nov 01 22:58:41.813859 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:41.813819 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-5xlgp"] Nov 01 22:58:43.750304 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:43.750264 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00381713-efff-4f16-8fc8-af879b07e517" path="/var/lib/kubelet/pods/00381713-efff-4f16-8fc8-af879b07e517/volumes" Nov 01 22:58:43.750869 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:43.750765 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="831636af-90aa-41a1-b03f-561c0462ab4e" path="/var/lib/kubelet/pods/831636af-90aa-41a1-b03f-561c0462ab4e/volumes" Nov 01 22:58:43.751278 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:43.751260 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee08f7f3-27a6-44fc-9099-dbe8459e6b12" path="/var/lib/kubelet/pods/ee08f7f3-27a6-44fc-9099-dbe8459e6b12/volumes" Nov 01 22:58:46.915680 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:46.915645 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0e685d-784cb6758c-4nwqk"] Nov 01 22:58:46.927711 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:46.927678 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0e685d-6f4746bcb7-l65sf"] Nov 01 22:58:46.981338 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:58:46.981276 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:58:51.916091 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:51.916043 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-0e685d-784cb6758c-4nwqk" podUID="8b86735b-2e85-4dc0-aff1-180a64cc76d0" containerName="payload-processing" containerID="cri-o://2b40ca1600c210fa01c93a2b9f3d74a781a2e7c2ba575bd15e3ced40059049cc" gracePeriod=25 Nov 01 22:58:51.928821 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:51.928774 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e685d-6f4746bcb7-l65sf" podUID="f027946a-f04e-4d9a-babc-bc81a506c5a4" containerName="payload-pre-processing" containerID="cri-o://410b3c9029c1da2443fed8159242888d7373a69852645b3316787ae030ece2b7" gracePeriod=25 Nov 01 22:58:52.654674 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:52.654631 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e685d-6f4746bcb7-l65sf" podUID="f027946a-f04e-4d9a-babc-bc81a506c5a4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:58:52.654884 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:52.654712 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-0e685d-784cb6758c-4nwqk" podUID="8b86735b-2e85-4dc0-aff1-180a64cc76d0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:58:54.744069 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:58:54.744038 5011 scope.go:122] "RemoveContainer" containerID="b8ac5cc4ebe6f774ea9658e32cc36197dd8fbf9b7883b18f83a516c611a2afb9" Nov 01 22:59:00.869934 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:00.869883 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0e685d-649786988c-nx4df"] Nov 01 22:59:00.870793 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:00.870762 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e685d-649786988c-nx4df" podUID="84b76836-25e9-4bb4-9e99-fd9aa116c30e" containerName="maas-api" containerID="cri-o://dd70edb64d035321229d06b783ae599e8aa24bff635931f3f4b77a68098cab07" gracePeriod=30 Nov 01 22:59:01.364301 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:01.362621 5011 generic.go:356] "Generic (PLEG): container finished" podID="84b76836-25e9-4bb4-9e99-fd9aa116c30e" containerID="dd70edb64d035321229d06b783ae599e8aa24bff635931f3f4b77a68098cab07" exitCode=0 Nov 01 22:59:01.364301 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:01.362801 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e685d-649786988c-nx4df" event={"ID":"84b76836-25e9-4bb4-9e99-fd9aa116c30e","Type":"ContainerDied","Data":"dd70edb64d035321229d06b783ae599e8aa24bff635931f3f4b77a68098cab07"} Nov 01 22:59:01.465270 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:01.465237 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e685d-649786988c-nx4df" Nov 01 22:59:01.549335 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:01.549302 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84b76836-25e9-4bb4-9e99-fd9aa116c30e-kube-api-access-vltqc\" (UniqueName: \"kubernetes.io/projected/84b76836-25e9-4bb4-9e99-fd9aa116c30e-kube-api-access-vltqc\") pod \"84b76836-25e9-4bb4-9e99-fd9aa116c30e\" (UID: \"84b76836-25e9-4bb4-9e99-fd9aa116c30e\") " Nov 01 22:59:01.549568 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:01.549479 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84b76836-25e9-4bb4-9e99-fd9aa116c30e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/84b76836-25e9-4bb4-9e99-fd9aa116c30e-metrics-certs\") pod \"84b76836-25e9-4bb4-9e99-fd9aa116c30e\" (UID: \"84b76836-25e9-4bb4-9e99-fd9aa116c30e\") " Nov 01 22:59:01.550275 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:01.549602 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84b76836-25e9-4bb4-9e99-fd9aa116c30e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b76836-25e9-4bb4-9e99-fd9aa116c30e-openshift-service-ca\") pod \"84b76836-25e9-4bb4-9e99-fd9aa116c30e\" (UID: \"84b76836-25e9-4bb4-9e99-fd9aa116c30e\") " Nov 01 22:59:01.550275 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:01.549803 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84b76836-25e9-4bb4-9e99-fd9aa116c30e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84b76836-25e9-4bb4-9e99-fd9aa116c30e-maas-api-tls\") pod \"84b76836-25e9-4bb4-9e99-fd9aa116c30e\" (UID: \"84b76836-25e9-4bb4-9e99-fd9aa116c30e\") " Nov 01 22:59:01.550275 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:01.550009 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b76836-25e9-4bb4-9e99-fd9aa116c30e-openshift-service-ca" pod "84b76836-25e9-4bb4-9e99-fd9aa116c30e" (UID: "84b76836-25e9-4bb4-9e99-fd9aa116c30e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:59:01.551112 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:01.551087 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b76836-25e9-4bb4-9e99-fd9aa116c30e-openshift-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:01.552272 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:01.552199 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84b76836-25e9-4bb4-9e99-fd9aa116c30e-maas-api-tls" pod "84b76836-25e9-4bb4-9e99-fd9aa116c30e" (UID: "84b76836-25e9-4bb4-9e99-fd9aa116c30e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:59:01.552406 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:01.552315 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b76836-25e9-4bb4-9e99-fd9aa116c30e-kube-api-access-vltqc" pod "84b76836-25e9-4bb4-9e99-fd9aa116c30e" (UID: "84b76836-25e9-4bb4-9e99-fd9aa116c30e"). InnerVolumeSpecName "kube-api-access-vltqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:59:01.552406 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:01.552322 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84b76836-25e9-4bb4-9e99-fd9aa116c30e-metrics-certs" pod "84b76836-25e9-4bb4-9e99-fd9aa116c30e" (UID: "84b76836-25e9-4bb4-9e99-fd9aa116c30e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:59:01.671912 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:01.654601 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84b76836-25e9-4bb4-9e99-fd9aa116c30e-metrics-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:01.671912 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:01.654643 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84b76836-25e9-4bb4-9e99-fd9aa116c30e-maas-api-tls\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:01.671912 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:01.654658 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vltqc\" (UniqueName: \"kubernetes.io/projected/84b76836-25e9-4bb4-9e99-fd9aa116c30e-kube-api-access-vltqc\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:02.406499 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:02.403698 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e685d-649786988c-nx4df" event={"ID":"84b76836-25e9-4bb4-9e99-fd9aa116c30e","Type":"ContainerDied","Data":"f2a94c3da07e54d10d8e041df91788e08e7bfd629a54f2c350386808c3121d0e"} Nov 01 22:59:02.406499 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:02.403779 5011 scope.go:122] "RemoveContainer" containerID="dd70edb64d035321229d06b783ae599e8aa24bff635931f3f4b77a68098cab07" Nov 01 22:59:02.406499 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:02.404007 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0e685d-649786988c-nx4df" Nov 01 22:59:02.431087 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:02.431054 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0e685d-649786988c-nx4df"] Nov 01 22:59:02.437675 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:02.437644 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0e685d-649786988c-nx4df"] Nov 01 22:59:02.656659 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:02.656574 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-0e685d-784cb6758c-4nwqk" podUID="8b86735b-2e85-4dc0-aff1-180a64cc76d0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:59:02.656659 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:02.656592 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e685d-6f4746bcb7-l65sf" podUID="f027946a-f04e-4d9a-babc-bc81a506c5a4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:59:03.749084 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:03.749046 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84b76836-25e9-4bb4-9e99-fd9aa116c30e" path="/var/lib/kubelet/pods/84b76836-25e9-4bb4-9e99-fd9aa116c30e/volumes" Nov 01 22:59:12.257306 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:12.257278 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0e685d-784cb6758c-4nwqk" Nov 01 22:59:12.277391 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:12.277360 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e685d-6f4746bcb7-l65sf" Nov 01 22:59:12.389904 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:12.389865 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b86735b-2e85-4dc0-aff1-180a64cc76d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b86735b-2e85-4dc0-aff1-180a64cc76d0-plugins-config-volume\") pod \"8b86735b-2e85-4dc0-aff1-180a64cc76d0\" (UID: \"8b86735b-2e85-4dc0-aff1-180a64cc76d0\") " Nov 01 22:59:12.389904 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:12.389910 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b86735b-2e85-4dc0-aff1-180a64cc76d0-kube-api-access-445fp\" (UniqueName: \"kubernetes.io/projected/8b86735b-2e85-4dc0-aff1-180a64cc76d0-kube-api-access-445fp\") pod \"8b86735b-2e85-4dc0-aff1-180a64cc76d0\" (UID: \"8b86735b-2e85-4dc0-aff1-180a64cc76d0\") " Nov 01 22:59:12.390129 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:12.390051 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f027946a-f04e-4d9a-babc-bc81a506c5a4-kube-api-access-gsdvd\" (UniqueName: \"kubernetes.io/projected/f027946a-f04e-4d9a-babc-bc81a506c5a4-kube-api-access-gsdvd\") pod \"f027946a-f04e-4d9a-babc-bc81a506c5a4\" (UID: \"f027946a-f04e-4d9a-babc-bc81a506c5a4\") " Nov 01 22:59:12.390164 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:12.390150 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f027946a-f04e-4d9a-babc-bc81a506c5a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f027946a-f04e-4d9a-babc-bc81a506c5a4-plugins-config-volume\") pod \"f027946a-f04e-4d9a-babc-bc81a506c5a4\" (UID: \"f027946a-f04e-4d9a-babc-bc81a506c5a4\") " Nov 01 22:59:12.390337 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:12.390308 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b86735b-2e85-4dc0-aff1-180a64cc76d0-plugins-config-volume" pod "8b86735b-2e85-4dc0-aff1-180a64cc76d0" (UID: "8b86735b-2e85-4dc0-aff1-180a64cc76d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:59:12.390512 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:12.390486 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f027946a-f04e-4d9a-babc-bc81a506c5a4-plugins-config-volume" pod "f027946a-f04e-4d9a-babc-bc81a506c5a4" (UID: "f027946a-f04e-4d9a-babc-bc81a506c5a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:59:12.390867 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:12.390844 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f027946a-f04e-4d9a-babc-bc81a506c5a4-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:12.390932 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:12.390869 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b86735b-2e85-4dc0-aff1-180a64cc76d0-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:12.392384 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:12.392366 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b86735b-2e85-4dc0-aff1-180a64cc76d0-kube-api-access-445fp" pod "8b86735b-2e85-4dc0-aff1-180a64cc76d0" (UID: "8b86735b-2e85-4dc0-aff1-180a64cc76d0"). InnerVolumeSpecName "kube-api-access-445fp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:59:12.392429 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:12.392394 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f027946a-f04e-4d9a-babc-bc81a506c5a4-kube-api-access-gsdvd" pod "f027946a-f04e-4d9a-babc-bc81a506c5a4" (UID: "f027946a-f04e-4d9a-babc-bc81a506c5a4"). InnerVolumeSpecName "kube-api-access-gsdvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:59:12.490077 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:12.489981 5011 generic.go:356] "Generic (PLEG): container finished" podID="f027946a-f04e-4d9a-babc-bc81a506c5a4" containerID="410b3c9029c1da2443fed8159242888d7373a69852645b3316787ae030ece2b7" exitCode=0 Nov 01 22:59:12.490077 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:12.490025 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e685d-6f4746bcb7-l65sf" event={"ID":"f027946a-f04e-4d9a-babc-bc81a506c5a4","Type":"ContainerDied","Data":"410b3c9029c1da2443fed8159242888d7373a69852645b3316787ae030ece2b7"} Nov 01 22:59:12.490329 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:12.490094 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e685d-6f4746bcb7-l65sf" event={"ID":"f027946a-f04e-4d9a-babc-bc81a506c5a4","Type":"ContainerDied","Data":"708fd263f173a73f507cc2e596f5f31af8352594d9be8ea635126b9d517761c7"} Nov 01 22:59:12.490329 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:12.490038 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0e685d-6f4746bcb7-l65sf" Nov 01 22:59:12.490329 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:12.490123 5011 scope.go:122] "RemoveContainer" containerID="410b3c9029c1da2443fed8159242888d7373a69852645b3316787ae030ece2b7" Nov 01 22:59:12.491784 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:12.491758 5011 generic.go:356] "Generic (PLEG): container finished" podID="8b86735b-2e85-4dc0-aff1-180a64cc76d0" containerID="2b40ca1600c210fa01c93a2b9f3d74a781a2e7c2ba575bd15e3ced40059049cc" exitCode=0 Nov 01 22:59:12.491960 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:12.491811 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0e685d-784cb6758c-4nwqk" Nov 01 22:59:12.491960 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:12.491858 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0e685d-784cb6758c-4nwqk" event={"ID":"8b86735b-2e85-4dc0-aff1-180a64cc76d0","Type":"ContainerDied","Data":"2b40ca1600c210fa01c93a2b9f3d74a781a2e7c2ba575bd15e3ced40059049cc"} Nov 01 22:59:12.491960 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:12.491916 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0e685d-784cb6758c-4nwqk" event={"ID":"8b86735b-2e85-4dc0-aff1-180a64cc76d0","Type":"ContainerDied","Data":"07c3c01a3f9b2771bc44cb5c63c46133495f347b6014d1ca54c9ad5053dcc791"} Nov 01 22:59:12.492109 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:12.492013 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-445fp\" (UniqueName: \"kubernetes.io/projected/8b86735b-2e85-4dc0-aff1-180a64cc76d0-kube-api-access-445fp\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:12.492109 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:12.492038 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsdvd\" (UniqueName: \"kubernetes.io/projected/f027946a-f04e-4d9a-babc-bc81a506c5a4-kube-api-access-gsdvd\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:12.504484 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:12.504462 5011 scope.go:122] "RemoveContainer" containerID="410b3c9029c1da2443fed8159242888d7373a69852645b3316787ae030ece2b7" Nov 01 22:59:12.504785 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:59:12.504765 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"410b3c9029c1da2443fed8159242888d7373a69852645b3316787ae030ece2b7\": container with ID starting with 410b3c9029c1da2443fed8159242888d7373a69852645b3316787ae030ece2b7 not found: ID does not exist" containerID="410b3c9029c1da2443fed8159242888d7373a69852645b3316787ae030ece2b7" Nov 01 22:59:12.504847 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:12.504795 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"410b3c9029c1da2443fed8159242888d7373a69852645b3316787ae030ece2b7"} err="failed to get container status \"410b3c9029c1da2443fed8159242888d7373a69852645b3316787ae030ece2b7\": rpc error: code = NotFound desc = could not find container \"410b3c9029c1da2443fed8159242888d7373a69852645b3316787ae030ece2b7\": container with ID starting with 410b3c9029c1da2443fed8159242888d7373a69852645b3316787ae030ece2b7 not found: ID does not exist" Nov 01 22:59:12.504847 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:12.504812 5011 scope.go:122] "RemoveContainer" containerID="2b40ca1600c210fa01c93a2b9f3d74a781a2e7c2ba575bd15e3ced40059049cc" Nov 01 22:59:12.515598 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:12.515579 5011 scope.go:122] "RemoveContainer" containerID="2b40ca1600c210fa01c93a2b9f3d74a781a2e7c2ba575bd15e3ced40059049cc" Nov 01 22:59:12.515934 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:59:12.515908 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b40ca1600c210fa01c93a2b9f3d74a781a2e7c2ba575bd15e3ced40059049cc\": container with ID starting with 2b40ca1600c210fa01c93a2b9f3d74a781a2e7c2ba575bd15e3ced40059049cc not found: ID does not exist" containerID="2b40ca1600c210fa01c93a2b9f3d74a781a2e7c2ba575bd15e3ced40059049cc" Nov 01 22:59:12.516000 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:12.515943 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b40ca1600c210fa01c93a2b9f3d74a781a2e7c2ba575bd15e3ced40059049cc"} err="failed to get container status \"2b40ca1600c210fa01c93a2b9f3d74a781a2e7c2ba575bd15e3ced40059049cc\": rpc error: code = NotFound desc = could not find container \"2b40ca1600c210fa01c93a2b9f3d74a781a2e7c2ba575bd15e3ced40059049cc\": container with ID starting with 2b40ca1600c210fa01c93a2b9f3d74a781a2e7c2ba575bd15e3ced40059049cc not found: ID does not exist" Nov 01 22:59:12.523624 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:12.523590 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0e685d-6f4746bcb7-l65sf"] Nov 01 22:59:12.529219 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:12.529194 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0e685d-6f4746bcb7-l65sf"] Nov 01 22:59:12.543205 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:12.543165 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0e685d-784cb6758c-4nwqk"] Nov 01 22:59:12.547469 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:12.547407 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0e685d-784cb6758c-4nwqk"] Nov 01 22:59:13.752003 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:13.751957 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b86735b-2e85-4dc0-aff1-180a64cc76d0" path="/var/lib/kubelet/pods/8b86735b-2e85-4dc0-aff1-180a64cc76d0/volumes" Nov 01 22:59:13.752646 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:13.752630 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f027946a-f04e-4d9a-babc-bc81a506c5a4" path="/var/lib/kubelet/pods/f027946a-f04e-4d9a-babc-bc81a506c5a4/volumes" Nov 01 22:59:17.016119 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:59:17.016036 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 22:59:28.034013 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:28.033976 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:59:28.034540 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:28.034053 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" podUID="d7b301bf-5694-40b2-8789-07555de1ed0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:59:29.793741 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:29.793702 5011 generic.go:356] "Generic (PLEG): container finished" podID="89a5b17f-1549-470c-8968-01b013df9417" containerID="92768429b3f0b69587257e3da0d606a65e8b74fcfb37042a5aee1152e0dabe3e" exitCode=0 Nov 01 22:59:29.794387 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:29.794351 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" event={"ID":"89a5b17f-1549-470c-8968-01b013df9417","Type":"ContainerDied","Data":"92768429b3f0b69587257e3da0d606a65e8b74fcfb37042a5aee1152e0dabe3e"} Nov 01 22:59:29.794916 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:29.794898 5011 scope.go:122] "RemoveContainer" containerID="92768429b3f0b69587257e3da0d606a65e8b74fcfb37042a5aee1152e0dabe3e" Nov 01 22:59:29.796100 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:29.796076 5011 generic.go:356] "Generic (PLEG): container finished" podID="d7b301bf-5694-40b2-8789-07555de1ed0f" containerID="10375264c2c86cd17cb06109c5d4e5876ab93023af79de115915f4008c874104" exitCode=0 Nov 01 22:59:29.796195 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:29.796179 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" event={"ID":"d7b301bf-5694-40b2-8789-07555de1ed0f","Type":"ContainerDied","Data":"10375264c2c86cd17cb06109c5d4e5876ab93023af79de115915f4008c874104"} Nov 01 22:59:29.796549 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:29.796532 5011 scope.go:122] "RemoveContainer" containerID="10375264c2c86cd17cb06109c5d4e5876ab93023af79de115915f4008c874104" Nov 01 22:59:29.797105 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:29.797071 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:59:29.797280 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:29.797168 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:59:29.797280 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:29.797222 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:59:29.798488 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:29.798459 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:59:29.798601 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:29.798544 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:59:29.798666 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:29.798611 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 22:59:30.807167 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:30.807125 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" event={"ID":"d7b301bf-5694-40b2-8789-07555de1ed0f","Type":"ContainerStarted","Data":"5f3a28c4c719374f67c65e73ae62c9270d5d93664d4eef3f33b9914b7acb61e9"} Nov 01 22:59:30.809359 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:30.809327 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" event={"ID":"89a5b17f-1549-470c-8968-01b013df9417","Type":"ContainerStarted","Data":"a1466adfa6bed6f0484768245c91d5f57f158fd51a11e3bcc966c756dd195a39"} Nov 01 22:59:30.908142 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:30.908107 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:59:30.909196 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:30.909171 5011 patch_prober.go:29] interesting pod/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 01 22:59:30.909319 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:30.909229 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" podUID="89a5b17f-1549-470c-8968-01b013df9417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 01 22:59:31.401370 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:31.401332 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:59:31.401723 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:31.401661 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 01 22:59:31.401863 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:31.401773 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" podUID="d7b301bf-5694-40b2-8789-07555de1ed0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 01 22:59:31.908793 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:31.908749 5011 patch_prober.go:29] interesting pod/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 01 22:59:31.909235 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:31.908805 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" podUID="89a5b17f-1549-470c-8968-01b013df9417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 01 22:59:32.397864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:32.397224 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 01 22:59:32.397864 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:32.397310 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" podUID="d7b301bf-5694-40b2-8789-07555de1ed0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 01 22:59:32.908711 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:32.908678 5011 patch_prober.go:29] interesting pod/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 01 22:59:32.908920 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:32.908730 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" podUID="89a5b17f-1549-470c-8968-01b013df9417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 01 22:59:33.397316 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:33.397278 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 01 22:59:33.397496 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:33.397343 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" podUID="d7b301bf-5694-40b2-8789-07555de1ed0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 01 22:59:33.908262 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:33.908225 5011 patch_prober.go:29] interesting pod/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 01 22:59:33.908475 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:33.908285 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" podUID="89a5b17f-1549-470c-8968-01b013df9417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 01 22:59:34.397366 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:34.397328 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 01 22:59:34.397803 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:34.397389 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" podUID="d7b301bf-5694-40b2-8789-07555de1ed0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 01 22:59:34.908475 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:34.908442 5011 patch_prober.go:29] interesting pod/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 01 22:59:34.908747 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:34.908495 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" podUID="89a5b17f-1549-470c-8968-01b013df9417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 01 22:59:35.397593 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:35.397543 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 01 22:59:35.398133 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:35.397607 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" podUID="d7b301bf-5694-40b2-8789-07555de1ed0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 01 22:59:35.593916 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:35.589953 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:59:35.910699 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:35.908942 5011 patch_prober.go:29] interesting pod/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Nov 01 22:59:35.910699 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:35.909008 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" podUID="89a5b17f-1549-470c-8968-01b013df9417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Nov 01 22:59:36.407893 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:36.401965 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 01 22:59:36.407893 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:36.402028 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" podUID="d7b301bf-5694-40b2-8789-07555de1ed0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 01 22:59:36.912919 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:36.912886 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:59:36.914038 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:36.914015 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 22:59:37.402112 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:37.402081 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:59:37.402603 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:37.402578 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:59:37.403488 ip-10-0-142-238 kubenswrapper[5011]: I1101 22:59:37.403469 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 22:59:46.994084 ip-10-0-142-238 kubenswrapper[5011]: E1101 22:59:46.994039 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:00:00.302017 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.301978 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf"] Nov 01 23:00:00.303671 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.303643 5011 cpu_manager.go:414] "containerMap: removing container" podUID="8b86735b-2e85-4dc0-aff1-180a64cc76d0" containerName="payload-processing" Nov 01 23:00:00.303671 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.303669 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b86735b-2e85-4dc0-aff1-180a64cc76d0" containerName="payload-processing" Nov 01 23:00:00.304034 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.303715 5011 cpu_manager.go:414] "containerMap: removing container" podUID="f027946a-f04e-4d9a-babc-bc81a506c5a4" containerName="payload-pre-processing" Nov 01 23:00:00.304034 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.303724 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f027946a-f04e-4d9a-babc-bc81a506c5a4" containerName="payload-pre-processing" Nov 01 23:00:00.304034 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.303749 5011 cpu_manager.go:414] "containerMap: removing container" podUID="39d06520-7ba9-44c0-a35d-22cfce8aaf08" containerName="maas-api" Nov 01 23:00:00.304034 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.303757 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39d06520-7ba9-44c0-a35d-22cfce8aaf08" containerName="maas-api" Nov 01 23:00:00.304034 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.303772 5011 cpu_manager.go:414] "containerMap: removing container" podUID="84b76836-25e9-4bb4-9e99-fd9aa116c30e" containerName="maas-api" Nov 01 23:00:00.304034 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.303779 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84b76836-25e9-4bb4-9e99-fd9aa116c30e" containerName="maas-api" Nov 01 23:00:00.304034 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.303789 5011 cpu_manager.go:414] "containerMap: removing container" podUID="831636af-90aa-41a1-b03f-561c0462ab4e" containerName="istio-proxy" Nov 01 23:00:00.304034 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.303796 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="831636af-90aa-41a1-b03f-561c0462ab4e" containerName="istio-proxy" Nov 01 23:00:00.304034 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.303845 5011 cpu_manager.go:414] "containerMap: removing container" podUID="ee08f7f3-27a6-44fc-9099-dbe8459e6b12" containerName="istio-proxy" Nov 01 23:00:00.304034 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.303852 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee08f7f3-27a6-44fc-9099-dbe8459e6b12" containerName="istio-proxy" Nov 01 23:00:00.304034 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.303865 5011 cpu_manager.go:414] "containerMap: removing container" podUID="00381713-efff-4f16-8fc8-af879b07e517" containerName="istio-proxy" Nov 01 23:00:00.304034 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.303872 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00381713-efff-4f16-8fc8-af879b07e517" containerName="istio-proxy" Nov 01 23:00:00.304968 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.304058 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="f027946a-f04e-4d9a-babc-bc81a506c5a4" containerName="payload-pre-processing" Nov 01 23:00:00.304968 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.304076 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="39d06520-7ba9-44c0-a35d-22cfce8aaf08" containerName="maas-api" Nov 01 23:00:00.304968 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.304088 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="84b76836-25e9-4bb4-9e99-fd9aa116c30e" containerName="maas-api" Nov 01 23:00:00.304968 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.304097 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="831636af-90aa-41a1-b03f-561c0462ab4e" containerName="istio-proxy" Nov 01 23:00:00.304968 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.304114 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee08f7f3-27a6-44fc-9099-dbe8459e6b12" containerName="istio-proxy" Nov 01 23:00:00.304968 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.304125 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="00381713-efff-4f16-8fc8-af879b07e517" containerName="istio-proxy" Nov 01 23:00:00.304968 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.304140 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b86735b-2e85-4dc0-aff1-180a64cc76d0" containerName="payload-processing" Nov 01 23:00:00.312432 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.312408 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf" Nov 01 23:00:00.345528 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.345491 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:00:00.348455 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.348406 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf"] Nov 01 23:00:00.354926 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.354895 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh"] Nov 01 23:00:00.360807 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.360782 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh" Nov 01 23:00:00.369194 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.369152 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a79bda1-5574-4dd9-9751-06804463a700-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf\" (UID: \"9a79bda1-5574-4dd9-9751-06804463a700\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf" Nov 01 23:00:00.369359 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.369269 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr46s\" (UniqueName: \"kubernetes.io/projected/9a79bda1-5574-4dd9-9751-06804463a700-kube-api-access-pr46s\") pod \"maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf\" (UID: \"9a79bda1-5574-4dd9-9751-06804463a700\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf" Nov 01 23:00:00.371649 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.371623 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw"] Nov 01 23:00:00.397798 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.397761 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh"] Nov 01 23:00:00.397993 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.397925 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw" Nov 01 23:00:00.414951 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.414918 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw"] Nov 01 23:00:00.424283 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.422308 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl"] Nov 01 23:00:00.428893 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.428863 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl" Nov 01 23:00:00.431087 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.431053 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k"] Nov 01 23:00:00.454651 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.453100 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl"] Nov 01 23:00:00.454651 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.453438 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k" Nov 01 23:00:00.473473 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.473427 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k"] Nov 01 23:00:00.480887 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.480849 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2xcg\" (UniqueName: \"kubernetes.io/projected/206fbb7b-ceea-4b88-98b9-5359edccfc01-kube-api-access-j2xcg\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh\" (UID: \"206fbb7b-ceea-4b88-98b9-5359edccfc01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh" Nov 01 23:00:00.481064 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.480918 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr46s\" (UniqueName: \"kubernetes.io/projected/9a79bda1-5574-4dd9-9751-06804463a700-kube-api-access-pr46s\") pod \"maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf\" (UID: \"9a79bda1-5574-4dd9-9751-06804463a700\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf" Nov 01 23:00:00.481136 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.481092 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfjdk\" (UniqueName: \"kubernetes.io/projected/fb9f7e76-0ab7-465a-937f-f19fe48b82b0-kube-api-access-gfjdk\") pod \"maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl\" (UID: \"fb9f7e76-0ab7-465a-937f-f19fe48b82b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl" Nov 01 23:00:00.481194 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.481178 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78077924-93c1-4a66-a79a-bd391195e3ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw\" (UID: \"78077924-93c1-4a66-a79a-bd391195e3ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw" Nov 01 23:00:00.481357 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.481332 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb9f7e76-0ab7-465a-937f-f19fe48b82b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl\" (UID: \"fb9f7e76-0ab7-465a-937f-f19fe48b82b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl" Nov 01 23:00:00.481438 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.481380 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5bgt\" (UniqueName: \"kubernetes.io/projected/78077924-93c1-4a66-a79a-bd391195e3ce-kube-api-access-f5bgt\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw\" (UID: \"78077924-93c1-4a66-a79a-bd391195e3ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw" Nov 01 23:00:00.481497 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.481476 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/206fbb7b-ceea-4b88-98b9-5359edccfc01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh\" (UID: \"206fbb7b-ceea-4b88-98b9-5359edccfc01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh" Nov 01 23:00:00.481584 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.481565 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a79bda1-5574-4dd9-9751-06804463a700-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf\" (UID: \"9a79bda1-5574-4dd9-9751-06804463a700\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf" Nov 01 23:00:00.483143 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.483067 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a79bda1-5574-4dd9-9751-06804463a700-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf\" (UID: \"9a79bda1-5574-4dd9-9751-06804463a700\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf" Nov 01 23:00:00.502427 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.502393 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr46s\" (UniqueName: \"kubernetes.io/projected/9a79bda1-5574-4dd9-9751-06804463a700-kube-api-access-pr46s\") pod \"maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf\" (UID: \"9a79bda1-5574-4dd9-9751-06804463a700\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf" Nov 01 23:00:00.583351 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.583249 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2xcg\" (UniqueName: \"kubernetes.io/projected/206fbb7b-ceea-4b88-98b9-5359edccfc01-kube-api-access-j2xcg\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh\" (UID: \"206fbb7b-ceea-4b88-98b9-5359edccfc01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh" Nov 01 23:00:00.583351 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.583352 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6gkd\" (UniqueName: \"kubernetes.io/projected/bb7867dc-31eb-4d53-acb5-feabbdbea02a-kube-api-access-f6gkd\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k\" (UID: \"bb7867dc-31eb-4d53-acb5-feabbdbea02a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k" Nov 01 23:00:00.583606 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.583387 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfjdk\" (UniqueName: \"kubernetes.io/projected/fb9f7e76-0ab7-465a-937f-f19fe48b82b0-kube-api-access-gfjdk\") pod \"maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl\" (UID: \"fb9f7e76-0ab7-465a-937f-f19fe48b82b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl" Nov 01 23:00:00.583606 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.583481 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78077924-93c1-4a66-a79a-bd391195e3ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw\" (UID: \"78077924-93c1-4a66-a79a-bd391195e3ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw" Nov 01 23:00:00.583723 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.583616 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb7867dc-31eb-4d53-acb5-feabbdbea02a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k\" (UID: \"bb7867dc-31eb-4d53-acb5-feabbdbea02a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k" Nov 01 23:00:00.583784 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.583750 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb9f7e76-0ab7-465a-937f-f19fe48b82b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl\" (UID: \"fb9f7e76-0ab7-465a-937f-f19fe48b82b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl" Nov 01 23:00:00.583784 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.583777 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5bgt\" (UniqueName: \"kubernetes.io/projected/78077924-93c1-4a66-a79a-bd391195e3ce-kube-api-access-f5bgt\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw\" (UID: \"78077924-93c1-4a66-a79a-bd391195e3ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw" Nov 01 23:00:00.583939 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.583863 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/206fbb7b-ceea-4b88-98b9-5359edccfc01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh\" (UID: \"206fbb7b-ceea-4b88-98b9-5359edccfc01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh" Nov 01 23:00:00.584123 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.584098 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78077924-93c1-4a66-a79a-bd391195e3ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw\" (UID: \"78077924-93c1-4a66-a79a-bd391195e3ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw" Nov 01 23:00:00.584489 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.584462 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb9f7e76-0ab7-465a-937f-f19fe48b82b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl\" (UID: \"fb9f7e76-0ab7-465a-937f-f19fe48b82b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl" Nov 01 23:00:00.584580 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.584531 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/206fbb7b-ceea-4b88-98b9-5359edccfc01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh\" (UID: \"206fbb7b-ceea-4b88-98b9-5359edccfc01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh" Nov 01 23:00:00.597125 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.597085 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5bgt\" (UniqueName: \"kubernetes.io/projected/78077924-93c1-4a66-a79a-bd391195e3ce-kube-api-access-f5bgt\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw\" (UID: \"78077924-93c1-4a66-a79a-bd391195e3ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw" Nov 01 23:00:00.598273 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.598244 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2xcg\" (UniqueName: \"kubernetes.io/projected/206fbb7b-ceea-4b88-98b9-5359edccfc01-kube-api-access-j2xcg\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh\" (UID: \"206fbb7b-ceea-4b88-98b9-5359edccfc01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh" Nov 01 23:00:00.598561 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.598539 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfjdk\" (UniqueName: \"kubernetes.io/projected/fb9f7e76-0ab7-465a-937f-f19fe48b82b0-kube-api-access-gfjdk\") pod \"maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl\" (UID: \"fb9f7e76-0ab7-465a-937f-f19fe48b82b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl" Nov 01 23:00:00.629879 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.629794 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf" Nov 01 23:00:00.685850 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.685802 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb7867dc-31eb-4d53-acb5-feabbdbea02a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k\" (UID: \"bb7867dc-31eb-4d53-acb5-feabbdbea02a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k" Nov 01 23:00:00.686303 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.686284 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6gkd\" (UniqueName: \"kubernetes.io/projected/bb7867dc-31eb-4d53-acb5-feabbdbea02a-kube-api-access-f6gkd\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k\" (UID: \"bb7867dc-31eb-4d53-acb5-feabbdbea02a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k" Nov 01 23:00:00.686594 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.686560 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb7867dc-31eb-4d53-acb5-feabbdbea02a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k\" (UID: \"bb7867dc-31eb-4d53-acb5-feabbdbea02a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k" Nov 01 23:00:00.695902 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.695867 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6gkd\" (UniqueName: \"kubernetes.io/projected/bb7867dc-31eb-4d53-acb5-feabbdbea02a-kube-api-access-f6gkd\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k\" (UID: \"bb7867dc-31eb-4d53-acb5-feabbdbea02a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k" Nov 01 23:00:00.709253 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.709222 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh" Nov 01 23:00:00.729282 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.729248 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw" Nov 01 23:00:00.793952 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.793165 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl" Nov 01 23:00:00.802003 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.800848 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k" Nov 01 23:00:00.937258 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:00.937172 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf"] Nov 01 23:00:00.941087 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:00:00.941037 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a79bda1_5574_4dd9_9751_06804463a700.slice/crio-8c634d8a53ad563e22d4e8870349641751fc363eb6b667562e8f30059f666035 WatchSource:0}: Error finding container 8c634d8a53ad563e22d4e8870349641751fc363eb6b667562e8f30059f666035: Status 404 returned error can't find the container with id 8c634d8a53ad563e22d4e8870349641751fc363eb6b667562e8f30059f666035 Nov 01 23:00:01.063572 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:01.055913 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh"] Nov 01 23:00:01.076113 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:01.076061 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw"] Nov 01 23:00:01.210976 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:01.210919 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw" event={"ID":"78077924-93c1-4a66-a79a-bd391195e3ce","Type":"ContainerStarted","Data":"952c17e5d0b8d76095a72e99d0c00f1fed286db52ec72667030287b132bbbe58"} Nov 01 23:00:01.212508 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:01.212468 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf" event={"ID":"9a79bda1-5574-4dd9-9751-06804463a700","Type":"ContainerStarted","Data":"8c634d8a53ad563e22d4e8870349641751fc363eb6b667562e8f30059f666035"} Nov 01 23:00:01.214157 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:01.214130 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh" event={"ID":"206fbb7b-ceea-4b88-98b9-5359edccfc01","Type":"ContainerStarted","Data":"1797a6230a9a4c4a11e7aa8bccaf11584fa4a0d27b417e6ae8b202933c8b37e5"} Nov 01 23:00:01.320433 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:01.320401 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl"] Nov 01 23:00:01.335330 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:01.335301 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k"] Nov 01 23:00:01.346993 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:00:01.346919 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb9f7e76_0ab7_465a_937f_f19fe48b82b0.slice/crio-23b938f6d18740c26c6ac101cc9acee77e3d9f654030092638db0b61ec284331 WatchSource:0}: Error finding container 23b938f6d18740c26c6ac101cc9acee77e3d9f654030092638db0b61ec284331: Status 404 returned error can't find the container with id 23b938f6d18740c26c6ac101cc9acee77e3d9f654030092638db0b61ec284331 Nov 01 23:00:01.348299 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:00:01.348270 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7867dc_31eb_4d53_acb5_feabbdbea02a.slice/crio-4fcace48832398ecc0392e7c22d7a5ff615af69642fdc32b7b5363fc76b0dd4d WatchSource:0}: Error finding container 4fcace48832398ecc0392e7c22d7a5ff615af69642fdc32b7b5363fc76b0dd4d: Status 404 returned error can't find the container with id 4fcace48832398ecc0392e7c22d7a5ff615af69642fdc32b7b5363fc76b0dd4d Nov 01 23:00:02.225451 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:02.225357 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw" event={"ID":"78077924-93c1-4a66-a79a-bd391195e3ce","Type":"ContainerStarted","Data":"ea4a98dc7b77a376bfe8233ba09bc811112d4b9310c2d78d1b8a0821fa02587f"} Nov 01 23:00:02.228268 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:02.228237 5011 generic.go:356] "Generic (PLEG): container finished" podID="9a79bda1-5574-4dd9-9751-06804463a700" containerID="40d1219969e2603724b748e11eac7a6b86487c4e3e82ada727699560a9411518" exitCode=60 Nov 01 23:00:02.228508 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:02.228292 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf" event={"ID":"9a79bda1-5574-4dd9-9751-06804463a700","Type":"ContainerDied","Data":"40d1219969e2603724b748e11eac7a6b86487c4e3e82ada727699560a9411518"} Nov 01 23:00:02.228662 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:02.228641 5011 scope.go:122] "RemoveContainer" containerID="40d1219969e2603724b748e11eac7a6b86487c4e3e82ada727699560a9411518" Nov 01 23:00:02.230602 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:02.230567 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh" event={"ID":"206fbb7b-ceea-4b88-98b9-5359edccfc01","Type":"ContainerStarted","Data":"5a255050294311e8a946850a5846fba36c294cce4f7050221836e662f7aa77e8"} Nov 01 23:00:02.232954 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:02.232923 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl" event={"ID":"fb9f7e76-0ab7-465a-937f-f19fe48b82b0","Type":"ContainerStarted","Data":"99aaa381bec681e1ccd1ae0e74411b7f747bd5ce4a12cbe69ebd913747b724b0"} Nov 01 23:00:02.233072 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:02.232968 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl" event={"ID":"fb9f7e76-0ab7-465a-937f-f19fe48b82b0","Type":"ContainerStarted","Data":"23b938f6d18740c26c6ac101cc9acee77e3d9f654030092638db0b61ec284331"} Nov 01 23:00:02.234679 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:02.234659 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k" event={"ID":"bb7867dc-31eb-4d53-acb5-feabbdbea02a","Type":"ContainerStarted","Data":"53e818ae2dbb51df0ce958366571685c9b923e5b06cad1f6b9e25807e7aa9b10"} Nov 01 23:00:02.234770 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:02.234694 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k" event={"ID":"bb7867dc-31eb-4d53-acb5-feabbdbea02a","Type":"ContainerStarted","Data":"4fcace48832398ecc0392e7c22d7a5ff615af69642fdc32b7b5363fc76b0dd4d"} Nov 01 23:00:02.246884 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:02.245439 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw" podStartSLOduration=2.245425526 podStartE2EDuration="2.245425526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:02.242539364 +0000 UTC m=+2170.647129998" watchObservedRunningTime="2026-11-01 23:00:02.245425526 +0000 UTC m=+2170.650016159" Nov 01 23:00:02.259913 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:02.259867 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl" podStartSLOduration=2.259813862 podStartE2EDuration="2.259813862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:02.259353362 +0000 UTC m=+2170.663943992" watchObservedRunningTime="2026-11-01 23:00:02.259813862 +0000 UTC m=+2170.664404493" Nov 01 23:00:02.279711 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:02.279654 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k" podStartSLOduration=2.279637936 podStartE2EDuration="2.279637936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:02.275433725 +0000 UTC m=+2170.680024358" watchObservedRunningTime="2026-11-01 23:00:02.279637936 +0000 UTC m=+2170.684228567" Nov 01 23:00:02.297730 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:02.297680 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh" podStartSLOduration=2.297664593 podStartE2EDuration="2.297664593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:02.293312579 +0000 UTC m=+2170.697903223" watchObservedRunningTime="2026-11-01 23:00:02.297664593 +0000 UTC m=+2170.702255225" Nov 01 23:00:03.246403 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:03.246367 5011 generic.go:356] "Generic (PLEG): container finished" podID="bb7867dc-31eb-4d53-acb5-feabbdbea02a" containerID="53e818ae2dbb51df0ce958366571685c9b923e5b06cad1f6b9e25807e7aa9b10" exitCode=60 Nov 01 23:00:03.246890 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:03.246510 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k" event={"ID":"bb7867dc-31eb-4d53-acb5-feabbdbea02a","Type":"ContainerDied","Data":"53e818ae2dbb51df0ce958366571685c9b923e5b06cad1f6b9e25807e7aa9b10"} Nov 01 23:00:03.246950 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:03.246899 5011 scope.go:122] "RemoveContainer" containerID="53e818ae2dbb51df0ce958366571685c9b923e5b06cad1f6b9e25807e7aa9b10" Nov 01 23:00:03.248586 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:03.248563 5011 generic.go:356] "Generic (PLEG): container finished" podID="78077924-93c1-4a66-a79a-bd391195e3ce" containerID="ea4a98dc7b77a376bfe8233ba09bc811112d4b9310c2d78d1b8a0821fa02587f" exitCode=60 Nov 01 23:00:03.248692 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:03.248638 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw" event={"ID":"78077924-93c1-4a66-a79a-bd391195e3ce","Type":"ContainerDied","Data":"ea4a98dc7b77a376bfe8233ba09bc811112d4b9310c2d78d1b8a0821fa02587f"} Nov 01 23:00:03.249289 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:03.249053 5011 scope.go:122] "RemoveContainer" containerID="ea4a98dc7b77a376bfe8233ba09bc811112d4b9310c2d78d1b8a0821fa02587f" Nov 01 23:00:03.254174 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:03.251578 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf" event={"ID":"9a79bda1-5574-4dd9-9751-06804463a700","Type":"ContainerStarted","Data":"bc91e20cef261d22c42200ee3550514ba75508730ec41ee13260d79874b0fc37"} Nov 01 23:00:03.254174 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:03.253785 5011 generic.go:356] "Generic (PLEG): container finished" podID="206fbb7b-ceea-4b88-98b9-5359edccfc01" containerID="5a255050294311e8a946850a5846fba36c294cce4f7050221836e662f7aa77e8" exitCode=60 Nov 01 23:00:03.254174 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:03.253872 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh" event={"ID":"206fbb7b-ceea-4b88-98b9-5359edccfc01","Type":"ContainerDied","Data":"5a255050294311e8a946850a5846fba36c294cce4f7050221836e662f7aa77e8"} Nov 01 23:00:03.254174 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:03.254152 5011 scope.go:122] "RemoveContainer" containerID="5a255050294311e8a946850a5846fba36c294cce4f7050221836e662f7aa77e8" Nov 01 23:00:03.255750 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:03.255727 5011 generic.go:356] "Generic (PLEG): container finished" podID="fb9f7e76-0ab7-465a-937f-f19fe48b82b0" containerID="99aaa381bec681e1ccd1ae0e74411b7f747bd5ce4a12cbe69ebd913747b724b0" exitCode=60 Nov 01 23:00:03.255902 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:03.255776 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl" event={"ID":"fb9f7e76-0ab7-465a-937f-f19fe48b82b0","Type":"ContainerDied","Data":"99aaa381bec681e1ccd1ae0e74411b7f747bd5ce4a12cbe69ebd913747b724b0"} Nov 01 23:00:03.256078 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:03.256065 5011 scope.go:122] "RemoveContainer" containerID="99aaa381bec681e1ccd1ae0e74411b7f747bd5ce4a12cbe69ebd913747b724b0" Nov 01 23:00:03.334926 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:03.334871 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf" podStartSLOduration=3.33485734 podStartE2EDuration="3.33485734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:03.33089276 +0000 UTC m=+2171.735483384" watchObservedRunningTime="2026-11-01 23:00:03.33485734 +0000 UTC m=+2171.739447969" Nov 01 23:00:04.267140 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:04.267105 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k" event={"ID":"bb7867dc-31eb-4d53-acb5-feabbdbea02a","Type":"ContainerStarted","Data":"846e81ca1b79d2a0f6944ce16f8de9bb489de35f6194999b407c820016c91687"} Nov 01 23:00:04.269210 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:04.269182 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw" event={"ID":"78077924-93c1-4a66-a79a-bd391195e3ce","Type":"ContainerStarted","Data":"4bcc209c9c555621297e42a7ae980cc64cc1afd84fffa1ffaa337819b91ecc2d"} Nov 01 23:00:04.271311 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:04.271281 5011 generic.go:356] "Generic (PLEG): container finished" podID="9a79bda1-5574-4dd9-9751-06804463a700" containerID="bc91e20cef261d22c42200ee3550514ba75508730ec41ee13260d79874b0fc37" exitCode=60 Nov 01 23:00:04.271444 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:04.271355 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf" event={"ID":"9a79bda1-5574-4dd9-9751-06804463a700","Type":"ContainerDied","Data":"bc91e20cef261d22c42200ee3550514ba75508730ec41ee13260d79874b0fc37"} Nov 01 23:00:04.271444 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:04.271407 5011 scope.go:122] "RemoveContainer" containerID="40d1219969e2603724b748e11eac7a6b86487c4e3e82ada727699560a9411518" Nov 01 23:00:04.271595 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:04.271580 5011 scope.go:122] "RemoveContainer" containerID="bc91e20cef261d22c42200ee3550514ba75508730ec41ee13260d79874b0fc37" Nov 01 23:00:04.272044 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:00:04.272014 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf_odh-ai-gateway-infra(9a79bda1-5574-4dd9-9751-06804463a700)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf" podUID="9a79bda1-5574-4dd9-9751-06804463a700" Nov 01 23:00:04.273532 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:04.273511 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh" event={"ID":"206fbb7b-ceea-4b88-98b9-5359edccfc01","Type":"ContainerStarted","Data":"63537c937a2d32e38f6c6bc778499a10f161703d84228d103a0416ac04ff16b6"} Nov 01 23:00:04.275423 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:04.275400 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl" event={"ID":"fb9f7e76-0ab7-465a-937f-f19fe48b82b0","Type":"ContainerStarted","Data":"009e3fb7edad0889c3df8e930a1bb3b435a5232f54b360aac466b36a8ecf7ad0"} Nov 01 23:00:05.286705 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:05.286668 5011 generic.go:356] "Generic (PLEG): container finished" podID="78077924-93c1-4a66-a79a-bd391195e3ce" containerID="4bcc209c9c555621297e42a7ae980cc64cc1afd84fffa1ffaa337819b91ecc2d" exitCode=60 Nov 01 23:00:05.287212 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:05.286749 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw" event={"ID":"78077924-93c1-4a66-a79a-bd391195e3ce","Type":"ContainerDied","Data":"4bcc209c9c555621297e42a7ae980cc64cc1afd84fffa1ffaa337819b91ecc2d"} Nov 01 23:00:05.287212 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:05.286807 5011 scope.go:122] "RemoveContainer" containerID="ea4a98dc7b77a376bfe8233ba09bc811112d4b9310c2d78d1b8a0821fa02587f" Nov 01 23:00:05.287212 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:05.287143 5011 scope.go:122] "RemoveContainer" containerID="4bcc209c9c555621297e42a7ae980cc64cc1afd84fffa1ffaa337819b91ecc2d" Nov 01 23:00:05.287569 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:00:05.287545 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw_odh-ai-gateway-infra(78077924-93c1-4a66-a79a-bd391195e3ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw" podUID="78077924-93c1-4a66-a79a-bd391195e3ce" Nov 01 23:00:05.289122 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:05.289104 5011 scope.go:122] "RemoveContainer" containerID="bc91e20cef261d22c42200ee3550514ba75508730ec41ee13260d79874b0fc37" Nov 01 23:00:05.289506 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:00:05.289487 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf_odh-ai-gateway-infra(9a79bda1-5574-4dd9-9751-06804463a700)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf" podUID="9a79bda1-5574-4dd9-9751-06804463a700" Nov 01 23:00:05.290693 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:05.290674 5011 generic.go:356] "Generic (PLEG): container finished" podID="206fbb7b-ceea-4b88-98b9-5359edccfc01" containerID="63537c937a2d32e38f6c6bc778499a10f161703d84228d103a0416ac04ff16b6" exitCode=60 Nov 01 23:00:05.290791 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:05.290754 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh" event={"ID":"206fbb7b-ceea-4b88-98b9-5359edccfc01","Type":"ContainerDied","Data":"63537c937a2d32e38f6c6bc778499a10f161703d84228d103a0416ac04ff16b6"} Nov 01 23:00:05.291108 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:05.291089 5011 scope.go:122] "RemoveContainer" containerID="63537c937a2d32e38f6c6bc778499a10f161703d84228d103a0416ac04ff16b6" Nov 01 23:00:05.291476 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:00:05.291453 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh_odh-ai-gateway-infra(206fbb7b-ceea-4b88-98b9-5359edccfc01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh" podUID="206fbb7b-ceea-4b88-98b9-5359edccfc01" Nov 01 23:00:05.292743 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:05.292723 5011 generic.go:356] "Generic (PLEG): container finished" podID="fb9f7e76-0ab7-465a-937f-f19fe48b82b0" containerID="009e3fb7edad0889c3df8e930a1bb3b435a5232f54b360aac466b36a8ecf7ad0" exitCode=60 Nov 01 23:00:05.292825 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:05.292804 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl" event={"ID":"fb9f7e76-0ab7-465a-937f-f19fe48b82b0","Type":"ContainerDied","Data":"009e3fb7edad0889c3df8e930a1bb3b435a5232f54b360aac466b36a8ecf7ad0"} Nov 01 23:00:05.293044 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:05.293026 5011 scope.go:122] "RemoveContainer" containerID="009e3fb7edad0889c3df8e930a1bb3b435a5232f54b360aac466b36a8ecf7ad0" Nov 01 23:00:05.293329 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:00:05.293305 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl_odh-ai-gateway-infra(fb9f7e76-0ab7-465a-937f-f19fe48b82b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl" podUID="fb9f7e76-0ab7-465a-937f-f19fe48b82b0" Nov 01 23:00:05.295361 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:05.295341 5011 generic.go:356] "Generic (PLEG): container finished" podID="bb7867dc-31eb-4d53-acb5-feabbdbea02a" containerID="846e81ca1b79d2a0f6944ce16f8de9bb489de35f6194999b407c820016c91687" exitCode=60 Nov 01 23:00:05.295454 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:05.295429 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k" event={"ID":"bb7867dc-31eb-4d53-acb5-feabbdbea02a","Type":"ContainerDied","Data":"846e81ca1b79d2a0f6944ce16f8de9bb489de35f6194999b407c820016c91687"} Nov 01 23:00:05.295728 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:05.295712 5011 scope.go:122] "RemoveContainer" containerID="846e81ca1b79d2a0f6944ce16f8de9bb489de35f6194999b407c820016c91687" Nov 01 23:00:05.296074 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:00:05.296054 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k_odh-ai-gateway-infra(bb7867dc-31eb-4d53-acb5-feabbdbea02a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k" podUID="bb7867dc-31eb-4d53-acb5-feabbdbea02a" Nov 01 23:00:05.302179 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:05.302160 5011 scope.go:122] "RemoveContainer" containerID="5a255050294311e8a946850a5846fba36c294cce4f7050221836e662f7aa77e8" Nov 01 23:00:05.319821 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:05.318325 5011 scope.go:122] "RemoveContainer" containerID="99aaa381bec681e1ccd1ae0e74411b7f747bd5ce4a12cbe69ebd913747b724b0" Nov 01 23:00:05.338304 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:05.338281 5011 scope.go:122] "RemoveContainer" containerID="53e818ae2dbb51df0ce958366571685c9b923e5b06cad1f6b9e25807e7aa9b10" Nov 01 23:00:06.305370 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:06.305342 5011 scope.go:122] "RemoveContainer" containerID="4bcc209c9c555621297e42a7ae980cc64cc1afd84fffa1ffaa337819b91ecc2d" Nov 01 23:00:06.305947 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:00:06.305800 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw_odh-ai-gateway-infra(78077924-93c1-4a66-a79a-bd391195e3ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw" podUID="78077924-93c1-4a66-a79a-bd391195e3ce" Nov 01 23:00:06.307228 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:06.307207 5011 scope.go:122] "RemoveContainer" containerID="63537c937a2d32e38f6c6bc778499a10f161703d84228d103a0416ac04ff16b6" Nov 01 23:00:06.307593 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:00:06.307570 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh_odh-ai-gateway-infra(206fbb7b-ceea-4b88-98b9-5359edccfc01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh" podUID="206fbb7b-ceea-4b88-98b9-5359edccfc01" Nov 01 23:00:06.308899 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:06.308880 5011 scope.go:122] "RemoveContainer" containerID="009e3fb7edad0889c3df8e930a1bb3b435a5232f54b360aac466b36a8ecf7ad0" Nov 01 23:00:06.309230 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:00:06.309205 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl_odh-ai-gateway-infra(fb9f7e76-0ab7-465a-937f-f19fe48b82b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl" podUID="fb9f7e76-0ab7-465a-937f-f19fe48b82b0" Nov 01 23:00:06.310730 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:06.310711 5011 scope.go:122] "RemoveContainer" containerID="846e81ca1b79d2a0f6944ce16f8de9bb489de35f6194999b407c820016c91687" Nov 01 23:00:06.311025 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:00:06.310998 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k_odh-ai-gateway-infra(bb7867dc-31eb-4d53-acb5-feabbdbea02a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k" podUID="bb7867dc-31eb-4d53-acb5-feabbdbea02a" Nov 01 23:00:14.273364 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:14.273326 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:00:16.739705 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:16.739679 5011 scope.go:122] "RemoveContainer" containerID="bc91e20cef261d22c42200ee3550514ba75508730ec41ee13260d79874b0fc37" Nov 01 23:00:16.740344 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:16.740319 5011 scope.go:122] "RemoveContainer" containerID="009e3fb7edad0889c3df8e930a1bb3b435a5232f54b360aac466b36a8ecf7ad0" Nov 01 23:00:16.740477 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:16.740414 5011 scope.go:122] "RemoveContainer" containerID="846e81ca1b79d2a0f6944ce16f8de9bb489de35f6194999b407c820016c91687" Nov 01 23:00:16.740927 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:16.740534 5011 scope.go:122] "RemoveContainer" containerID="63537c937a2d32e38f6c6bc778499a10f161703d84228d103a0416ac04ff16b6" Nov 01 23:00:16.986168 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:00:16.986132 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:00:17.529170 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:17.529121 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k" event={"ID":"bb7867dc-31eb-4d53-acb5-feabbdbea02a","Type":"ContainerStarted","Data":"54e21f8fe42762faef9b5ef18b1987315a91962187855db32cac6f77333098ef"} Nov 01 23:00:17.565676 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:17.565240 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf" event={"ID":"9a79bda1-5574-4dd9-9751-06804463a700","Type":"ContainerStarted","Data":"49fbe7acc129c0f454a05cac7a1af9f816415a594f90795dcfaf0d59d95bf892"} Nov 01 23:00:17.572468 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:17.572434 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh" event={"ID":"206fbb7b-ceea-4b88-98b9-5359edccfc01","Type":"ContainerStarted","Data":"d4c4944ece6fcb9bf78279032cd4460aba96eafca77c97988749a51a35710b7c"} Nov 01 23:00:17.580236 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:17.580197 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl" event={"ID":"fb9f7e76-0ab7-465a-937f-f19fe48b82b0","Type":"ContainerStarted","Data":"6a4c29110f4132300873db9c07cb88c3a3b099c7aa0b13ac2527cc5d72c166f8"} Nov 01 23:00:17.740224 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:17.740185 5011 scope.go:122] "RemoveContainer" containerID="4bcc209c9c555621297e42a7ae980cc64cc1afd84fffa1ffaa337819b91ecc2d" Nov 01 23:00:18.594447 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:18.594407 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw" event={"ID":"78077924-93c1-4a66-a79a-bd391195e3ce","Type":"ContainerStarted","Data":"243376e94c640389b7079ae852de8bde173bde6d70bc123b50a1e51d310317a9"} Nov 01 23:00:18.595512 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:18.595486 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k"] Nov 01 23:00:18.596858 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:18.596805 5011 generic.go:356] "Generic (PLEG): container finished" podID="9a79bda1-5574-4dd9-9751-06804463a700" containerID="49fbe7acc129c0f454a05cac7a1af9f816415a594f90795dcfaf0d59d95bf892" exitCode=60 Nov 01 23:00:18.596973 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:18.596900 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf" event={"ID":"9a79bda1-5574-4dd9-9751-06804463a700","Type":"ContainerDied","Data":"49fbe7acc129c0f454a05cac7a1af9f816415a594f90795dcfaf0d59d95bf892"} Nov 01 23:00:18.596973 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:18.596968 5011 scope.go:122] "RemoveContainer" containerID="bc91e20cef261d22c42200ee3550514ba75508730ec41ee13260d79874b0fc37" Nov 01 23:00:18.597332 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:18.597315 5011 scope.go:122] "RemoveContainer" containerID="49fbe7acc129c0f454a05cac7a1af9f816415a594f90795dcfaf0d59d95bf892" Nov 01 23:00:18.597674 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:00:18.597649 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf_odh-ai-gateway-infra(9a79bda1-5574-4dd9-9751-06804463a700)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf" podUID="9a79bda1-5574-4dd9-9751-06804463a700" Nov 01 23:00:18.658905 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:18.637412 5011 generic.go:356] "Generic (PLEG): container finished" podID="206fbb7b-ceea-4b88-98b9-5359edccfc01" containerID="d4c4944ece6fcb9bf78279032cd4460aba96eafca77c97988749a51a35710b7c" exitCode=60 Nov 01 23:00:18.658905 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:18.637538 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh" event={"ID":"206fbb7b-ceea-4b88-98b9-5359edccfc01","Type":"ContainerDied","Data":"d4c4944ece6fcb9bf78279032cd4460aba96eafca77c97988749a51a35710b7c"} Nov 01 23:00:18.658905 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:18.637606 5011 scope.go:122] "RemoveContainer" containerID="63537c937a2d32e38f6c6bc778499a10f161703d84228d103a0416ac04ff16b6" Nov 01 23:00:18.658905 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:18.638196 5011 scope.go:122] "RemoveContainer" containerID="d4c4944ece6fcb9bf78279032cd4460aba96eafca77c97988749a51a35710b7c" Nov 01 23:00:18.658905 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:00:18.638666 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh_odh-ai-gateway-infra(206fbb7b-ceea-4b88-98b9-5359edccfc01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh" podUID="206fbb7b-ceea-4b88-98b9-5359edccfc01" Nov 01 23:00:18.687286 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:18.683768 5011 generic.go:356] "Generic (PLEG): container finished" podID="fb9f7e76-0ab7-465a-937f-f19fe48b82b0" containerID="6a4c29110f4132300873db9c07cb88c3a3b099c7aa0b13ac2527cc5d72c166f8" exitCode=60 Nov 01 23:00:18.687286 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:18.684079 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl" event={"ID":"fb9f7e76-0ab7-465a-937f-f19fe48b82b0","Type":"ContainerDied","Data":"6a4c29110f4132300873db9c07cb88c3a3b099c7aa0b13ac2527cc5d72c166f8"} Nov 01 23:00:18.687286 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:18.684655 5011 scope.go:122] "RemoveContainer" containerID="6a4c29110f4132300873db9c07cb88c3a3b099c7aa0b13ac2527cc5d72c166f8" Nov 01 23:00:18.687286 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:00:18.685152 5011 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl_odh-ai-gateway-infra(fb9f7e76-0ab7-465a-937f-f19fe48b82b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl" podUID="fb9f7e76-0ab7-465a-937f-f19fe48b82b0" Nov 01 23:00:18.732271 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:18.732241 5011 scope.go:122] "RemoveContainer" containerID="009e3fb7edad0889c3df8e930a1bb3b435a5232f54b360aac466b36a8ecf7ad0" Nov 01 23:00:18.733167 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:18.732858 5011 generic.go:356] "Generic (PLEG): container finished" podID="bb7867dc-31eb-4d53-acb5-feabbdbea02a" containerID="54e21f8fe42762faef9b5ef18b1987315a91962187855db32cac6f77333098ef" exitCode=60 Nov 01 23:00:18.733167 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:18.733045 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k" event={"ID":"bb7867dc-31eb-4d53-acb5-feabbdbea02a","Type":"ContainerDied","Data":"54e21f8fe42762faef9b5ef18b1987315a91962187855db32cac6f77333098ef"} Nov 01 23:00:18.757937 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:18.757896 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf"] Nov 01 23:00:18.787113 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:18.784314 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-021714-7b74898bd7-s2gqb"] Nov 01 23:00:18.790253 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:18.789982 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh"] Nov 01 23:00:18.806196 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:18.800714 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl"] Nov 01 23:00:18.806196 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:18.803709 5011 scope.go:122] "RemoveContainer" containerID="846e81ca1b79d2a0f6944ce16f8de9bb489de35f6194999b407c820016c91687" Nov 01 23:00:19.300858 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:19.298798 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k" Nov 01 23:00:19.351674 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:19.351638 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb7867dc-31eb-4d53-acb5-feabbdbea02a-kube-api-access-f6gkd\" (UniqueName: \"kubernetes.io/projected/bb7867dc-31eb-4d53-acb5-feabbdbea02a-kube-api-access-f6gkd\") pod \"bb7867dc-31eb-4d53-acb5-feabbdbea02a\" (UID: \"bb7867dc-31eb-4d53-acb5-feabbdbea02a\") " Nov 01 23:00:19.351893 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:19.351694 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb7867dc-31eb-4d53-acb5-feabbdbea02a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb7867dc-31eb-4d53-acb5-feabbdbea02a-maas-api-service-ca\") pod \"bb7867dc-31eb-4d53-acb5-feabbdbea02a\" (UID: \"bb7867dc-31eb-4d53-acb5-feabbdbea02a\") " Nov 01 23:00:19.352189 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:19.352160 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb7867dc-31eb-4d53-acb5-feabbdbea02a-maas-api-service-ca" pod "bb7867dc-31eb-4d53-acb5-feabbdbea02a" (UID: "bb7867dc-31eb-4d53-acb5-feabbdbea02a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:19.352553 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:19.352535 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb7867dc-31eb-4d53-acb5-feabbdbea02a-maas-api-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:19.354449 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:19.354419 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb7867dc-31eb-4d53-acb5-feabbdbea02a-kube-api-access-f6gkd" pod "bb7867dc-31eb-4d53-acb5-feabbdbea02a" (UID: "bb7867dc-31eb-4d53-acb5-feabbdbea02a"). InnerVolumeSpecName "kube-api-access-f6gkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:19.454774 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:19.454681 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6gkd\" (UniqueName: \"kubernetes.io/projected/bb7867dc-31eb-4d53-acb5-feabbdbea02a-kube-api-access-f6gkd\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:19.636683 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:19.636642 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw"] Nov 01 23:00:19.744420 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:19.744336 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k" Nov 01 23:00:19.758960 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:19.758921 5011 generic.go:356] "Generic (PLEG): container finished" podID="78077924-93c1-4a66-a79a-bd391195e3ce" containerID="243376e94c640389b7079ae852de8bde173bde6d70bc123b50a1e51d310317a9" exitCode=60 Nov 01 23:00:19.762321 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:19.762293 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k" event={"ID":"bb7867dc-31eb-4d53-acb5-feabbdbea02a","Type":"ContainerDied","Data":"4fcace48832398ecc0392e7c22d7a5ff615af69642fdc32b7b5363fc76b0dd4d"} Nov 01 23:00:19.762438 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:19.762329 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw" event={"ID":"78077924-93c1-4a66-a79a-bd391195e3ce","Type":"ContainerDied","Data":"243376e94c640389b7079ae852de8bde173bde6d70bc123b50a1e51d310317a9"} Nov 01 23:00:19.762438 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:19.762375 5011 scope.go:122] "RemoveContainer" containerID="54e21f8fe42762faef9b5ef18b1987315a91962187855db32cac6f77333098ef" Nov 01 23:00:19.799860 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:19.794494 5011 scope.go:122] "RemoveContainer" containerID="4bcc209c9c555621297e42a7ae980cc64cc1afd84fffa1ffaa337819b91ecc2d" Nov 01 23:00:19.815406 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:19.811420 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k"] Nov 01 23:00:19.834862 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:19.829971 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-788d7b-29892900-lpv9k"] Nov 01 23:00:20.268309 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.268275 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl" Nov 01 23:00:20.350657 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.350629 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh" Nov 01 23:00:20.358097 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.358066 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf" Nov 01 23:00:20.361740 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.361662 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw" Nov 01 23:00:20.417856 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.412512 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a79bda1-5574-4dd9-9751-06804463a700-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a79bda1-5574-4dd9-9751-06804463a700-maas-api-service-ca\") pod \"9a79bda1-5574-4dd9-9751-06804463a700\" (UID: \"9a79bda1-5574-4dd9-9751-06804463a700\") " Nov 01 23:00:20.417856 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.412958 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a79bda1-5574-4dd9-9751-06804463a700-maas-api-service-ca" pod "9a79bda1-5574-4dd9-9751-06804463a700" (UID: "9a79bda1-5574-4dd9-9751-06804463a700"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:20.417856 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.413162 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/206fbb7b-ceea-4b88-98b9-5359edccfc01-kube-api-access-j2xcg\" (UniqueName: \"kubernetes.io/projected/206fbb7b-ceea-4b88-98b9-5359edccfc01-kube-api-access-j2xcg\") pod \"206fbb7b-ceea-4b88-98b9-5359edccfc01\" (UID: \"206fbb7b-ceea-4b88-98b9-5359edccfc01\") " Nov 01 23:00:20.417856 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.413565 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb9f7e76-0ab7-465a-937f-f19fe48b82b0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb9f7e76-0ab7-465a-937f-f19fe48b82b0-maas-api-service-ca\") pod \"fb9f7e76-0ab7-465a-937f-f19fe48b82b0\" (UID: \"fb9f7e76-0ab7-465a-937f-f19fe48b82b0\") " Nov 01 23:00:20.417856 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.413619 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78077924-93c1-4a66-a79a-bd391195e3ce-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78077924-93c1-4a66-a79a-bd391195e3ce-maas-api-service-ca\") pod \"78077924-93c1-4a66-a79a-bd391195e3ce\" (UID: \"78077924-93c1-4a66-a79a-bd391195e3ce\") " Nov 01 23:00:20.417856 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.413719 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb9f7e76-0ab7-465a-937f-f19fe48b82b0-kube-api-access-gfjdk\" (UniqueName: \"kubernetes.io/projected/fb9f7e76-0ab7-465a-937f-f19fe48b82b0-kube-api-access-gfjdk\") pod \"fb9f7e76-0ab7-465a-937f-f19fe48b82b0\" (UID: \"fb9f7e76-0ab7-465a-937f-f19fe48b82b0\") " Nov 01 23:00:20.417856 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.413758 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a79bda1-5574-4dd9-9751-06804463a700-kube-api-access-pr46s\" (UniqueName: \"kubernetes.io/projected/9a79bda1-5574-4dd9-9751-06804463a700-kube-api-access-pr46s\") pod \"9a79bda1-5574-4dd9-9751-06804463a700\" (UID: \"9a79bda1-5574-4dd9-9751-06804463a700\") " Nov 01 23:00:20.417856 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.414002 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/206fbb7b-ceea-4b88-98b9-5359edccfc01-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/206fbb7b-ceea-4b88-98b9-5359edccfc01-maas-api-service-ca\") pod \"206fbb7b-ceea-4b88-98b9-5359edccfc01\" (UID: \"206fbb7b-ceea-4b88-98b9-5359edccfc01\") " Nov 01 23:00:20.417856 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.414190 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78077924-93c1-4a66-a79a-bd391195e3ce-kube-api-access-f5bgt\" (UniqueName: \"kubernetes.io/projected/78077924-93c1-4a66-a79a-bd391195e3ce-kube-api-access-f5bgt\") pod \"78077924-93c1-4a66-a79a-bd391195e3ce\" (UID: \"78077924-93c1-4a66-a79a-bd391195e3ce\") " Nov 01 23:00:20.417856 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.414182 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb9f7e76-0ab7-465a-937f-f19fe48b82b0-maas-api-service-ca" pod "fb9f7e76-0ab7-465a-937f-f19fe48b82b0" (UID: "fb9f7e76-0ab7-465a-937f-f19fe48b82b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:20.417856 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.414672 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/206fbb7b-ceea-4b88-98b9-5359edccfc01-maas-api-service-ca" pod "206fbb7b-ceea-4b88-98b9-5359edccfc01" (UID: "206fbb7b-ceea-4b88-98b9-5359edccfc01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:20.417856 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.415007 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/206fbb7b-ceea-4b88-98b9-5359edccfc01-maas-api-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:20.417856 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.415030 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a79bda1-5574-4dd9-9751-06804463a700-maas-api-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:20.417856 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.415045 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb9f7e76-0ab7-465a-937f-f19fe48b82b0-maas-api-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:20.417856 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.415092 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78077924-93c1-4a66-a79a-bd391195e3ce-maas-api-service-ca" pod "78077924-93c1-4a66-a79a-bd391195e3ce" (UID: "78077924-93c1-4a66-a79a-bd391195e3ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:20.425900 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.420039 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/206fbb7b-ceea-4b88-98b9-5359edccfc01-kube-api-access-j2xcg" pod "206fbb7b-ceea-4b88-98b9-5359edccfc01" (UID: "206fbb7b-ceea-4b88-98b9-5359edccfc01"). InnerVolumeSpecName "kube-api-access-j2xcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:20.425900 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.425742 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a79bda1-5574-4dd9-9751-06804463a700-kube-api-access-pr46s" pod "9a79bda1-5574-4dd9-9751-06804463a700" (UID: "9a79bda1-5574-4dd9-9751-06804463a700"). InnerVolumeSpecName "kube-api-access-pr46s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:20.425900 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.425887 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb9f7e76-0ab7-465a-937f-f19fe48b82b0-kube-api-access-gfjdk" pod "fb9f7e76-0ab7-465a-937f-f19fe48b82b0" (UID: "fb9f7e76-0ab7-465a-937f-f19fe48b82b0"). InnerVolumeSpecName "kube-api-access-gfjdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:20.428585 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.428393 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78077924-93c1-4a66-a79a-bd391195e3ce-kube-api-access-f5bgt" pod "78077924-93c1-4a66-a79a-bd391195e3ce" (UID: "78077924-93c1-4a66-a79a-bd391195e3ce"). InnerVolumeSpecName "kube-api-access-f5bgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:20.517096 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.517053 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5bgt\" (UniqueName: \"kubernetes.io/projected/78077924-93c1-4a66-a79a-bd391195e3ce-kube-api-access-f5bgt\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:20.517096 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.517087 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2xcg\" (UniqueName: \"kubernetes.io/projected/206fbb7b-ceea-4b88-98b9-5359edccfc01-kube-api-access-j2xcg\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:20.517096 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.517097 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78077924-93c1-4a66-a79a-bd391195e3ce-maas-api-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:20.517096 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.517106 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfjdk\" (UniqueName: \"kubernetes.io/projected/fb9f7e76-0ab7-465a-937f-f19fe48b82b0-kube-api-access-gfjdk\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:20.517408 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.517115 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr46s\" (UniqueName: \"kubernetes.io/projected/9a79bda1-5574-4dd9-9751-06804463a700-kube-api-access-pr46s\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:20.788402 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.788360 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw" Nov 01 23:00:20.788402 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.788380 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw" event={"ID":"78077924-93c1-4a66-a79a-bd391195e3ce","Type":"ContainerDied","Data":"952c17e5d0b8d76095a72e99d0c00f1fed286db52ec72667030287b132bbbe58"} Nov 01 23:00:20.789014 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.788470 5011 scope.go:122] "RemoveContainer" containerID="243376e94c640389b7079ae852de8bde173bde6d70bc123b50a1e51d310317a9" Nov 01 23:00:20.791368 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.790921 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf" event={"ID":"9a79bda1-5574-4dd9-9751-06804463a700","Type":"ContainerDied","Data":"8c634d8a53ad563e22d4e8870349641751fc363eb6b667562e8f30059f666035"} Nov 01 23:00:20.791368 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.791057 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf" Nov 01 23:00:20.799586 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.799551 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh" event={"ID":"206fbb7b-ceea-4b88-98b9-5359edccfc01","Type":"ContainerDied","Data":"1797a6230a9a4c4a11e7aa8bccaf11584fa4a0d27b417e6ae8b202933c8b37e5"} Nov 01 23:00:20.799732 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.799668 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh" Nov 01 23:00:20.803178 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.803145 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl" event={"ID":"fb9f7e76-0ab7-465a-937f-f19fe48b82b0","Type":"ContainerDied","Data":"23b938f6d18740c26c6ac101cc9acee77e3d9f654030092638db0b61ec284331"} Nov 01 23:00:20.803295 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.803258 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl" Nov 01 23:00:20.810454 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.810424 5011 scope.go:122] "RemoveContainer" containerID="49fbe7acc129c0f454a05cac7a1af9f816415a594f90795dcfaf0d59d95bf892" Nov 01 23:00:20.825044 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.825016 5011 scope.go:122] "RemoveContainer" containerID="d4c4944ece6fcb9bf78279032cd4460aba96eafca77c97988749a51a35710b7c" Nov 01 23:00:20.830904 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.830874 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf"] Nov 01 23:00:20.841634 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.841593 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2f3705-29892900-bd9nf"] Nov 01 23:00:20.842126 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.842090 5011 scope.go:122] "RemoveContainer" containerID="6a4c29110f4132300873db9c07cb88c3a3b099c7aa0b13ac2527cc5d72c166f8" Nov 01 23:00:20.858760 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.858725 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw"] Nov 01 23:00:20.863807 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.863772 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892900-pgtdw"] Nov 01 23:00:20.884971 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.884902 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh"] Nov 01 23:00:20.895095 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.895062 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-49ab43-29892900-wfrvh"] Nov 01 23:00:20.919904 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.919819 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl"] Nov 01 23:00:20.929725 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:20.929680 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b48ab8-29892900-rgjtl"] Nov 01 23:00:21.750890 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:21.750857 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="206fbb7b-ceea-4b88-98b9-5359edccfc01" path="/var/lib/kubelet/pods/206fbb7b-ceea-4b88-98b9-5359edccfc01/volumes" Nov 01 23:00:21.751472 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:21.751450 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78077924-93c1-4a66-a79a-bd391195e3ce" path="/var/lib/kubelet/pods/78077924-93c1-4a66-a79a-bd391195e3ce/volumes" Nov 01 23:00:21.752096 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:21.752075 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a79bda1-5574-4dd9-9751-06804463a700" path="/var/lib/kubelet/pods/9a79bda1-5574-4dd9-9751-06804463a700/volumes" Nov 01 23:00:21.753058 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:21.753039 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb7867dc-31eb-4d53-acb5-feabbdbea02a" path="/var/lib/kubelet/pods/bb7867dc-31eb-4d53-acb5-feabbdbea02a/volumes" Nov 01 23:00:21.753443 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:21.753430 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb9f7e76-0ab7-465a-937f-f19fe48b82b0" path="/var/lib/kubelet/pods/fb9f7e76-0ab7-465a-937f-f19fe48b82b0/volumes" Nov 01 23:00:23.785546 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:23.785505 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-021714-7b74898bd7-s2gqb" podUID="02975ed3-18c1-49ee-8bae-e86edb4e0130" containerName="payload-processing" containerID="cri-o://fe589a25553a50e5ab341d7a39a63745bab2a6c58e2d8ea925ffad304e292bcf" gracePeriod=25 Nov 01 23:00:26.107492 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:26.107449 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-021714-7b74898bd7-s2gqb" podUID="02975ed3-18c1-49ee-8bae-e86edb4e0130" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:00:36.107499 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.107458 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-021714-7b74898bd7-s2gqb" podUID="02975ed3-18c1-49ee-8bae-e86edb4e0130" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:00:36.856446 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.856413 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf"] Nov 01 23:00:36.857243 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857225 5011 cpu_manager.go:414] "containerMap: removing container" podUID="9a79bda1-5574-4dd9-9751-06804463a700" containerName="cleanup" Nov 01 23:00:36.857315 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857244 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a79bda1-5574-4dd9-9751-06804463a700" containerName="cleanup" Nov 01 23:00:36.857315 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857256 5011 cpu_manager.go:414] "containerMap: removing container" podUID="206fbb7b-ceea-4b88-98b9-5359edccfc01" containerName="cleanup" Nov 01 23:00:36.857315 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857261 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="206fbb7b-ceea-4b88-98b9-5359edccfc01" containerName="cleanup" Nov 01 23:00:36.857315 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857267 5011 cpu_manager.go:414] "containerMap: removing container" podUID="78077924-93c1-4a66-a79a-bd391195e3ce" containerName="cleanup" Nov 01 23:00:36.857315 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857273 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78077924-93c1-4a66-a79a-bd391195e3ce" containerName="cleanup" Nov 01 23:00:36.857315 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857281 5011 cpu_manager.go:414] "containerMap: removing container" podUID="78077924-93c1-4a66-a79a-bd391195e3ce" containerName="cleanup" Nov 01 23:00:36.857315 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857285 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78077924-93c1-4a66-a79a-bd391195e3ce" containerName="cleanup" Nov 01 23:00:36.857315 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857292 5011 cpu_manager.go:414] "containerMap: removing container" podUID="bb7867dc-31eb-4d53-acb5-feabbdbea02a" containerName="cleanup" Nov 01 23:00:36.857315 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857296 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb7867dc-31eb-4d53-acb5-feabbdbea02a" containerName="cleanup" Nov 01 23:00:36.857315 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857308 5011 cpu_manager.go:414] "containerMap: removing container" podUID="bb7867dc-31eb-4d53-acb5-feabbdbea02a" containerName="cleanup" Nov 01 23:00:36.857315 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857313 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb7867dc-31eb-4d53-acb5-feabbdbea02a" containerName="cleanup" Nov 01 23:00:36.857648 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857334 5011 cpu_manager.go:414] "containerMap: removing container" podUID="78077924-93c1-4a66-a79a-bd391195e3ce" containerName="cleanup" Nov 01 23:00:36.857648 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857338 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78077924-93c1-4a66-a79a-bd391195e3ce" containerName="cleanup" Nov 01 23:00:36.857648 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857349 5011 cpu_manager.go:414] "containerMap: removing container" podUID="fb9f7e76-0ab7-465a-937f-f19fe48b82b0" containerName="cleanup" Nov 01 23:00:36.857648 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857353 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb9f7e76-0ab7-465a-937f-f19fe48b82b0" containerName="cleanup" Nov 01 23:00:36.857648 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857362 5011 cpu_manager.go:414] "containerMap: removing container" podUID="bb7867dc-31eb-4d53-acb5-feabbdbea02a" containerName="cleanup" Nov 01 23:00:36.857648 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857366 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb7867dc-31eb-4d53-acb5-feabbdbea02a" containerName="cleanup" Nov 01 23:00:36.857648 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857372 5011 cpu_manager.go:414] "containerMap: removing container" podUID="206fbb7b-ceea-4b88-98b9-5359edccfc01" containerName="cleanup" Nov 01 23:00:36.857648 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857376 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="206fbb7b-ceea-4b88-98b9-5359edccfc01" containerName="cleanup" Nov 01 23:00:36.857648 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857387 5011 cpu_manager.go:414] "containerMap: removing container" podUID="9a79bda1-5574-4dd9-9751-06804463a700" containerName="cleanup" Nov 01 23:00:36.857648 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857390 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a79bda1-5574-4dd9-9751-06804463a700" containerName="cleanup" Nov 01 23:00:36.857648 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857395 5011 cpu_manager.go:414] "containerMap: removing container" podUID="fb9f7e76-0ab7-465a-937f-f19fe48b82b0" containerName="cleanup" Nov 01 23:00:36.857648 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857399 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb9f7e76-0ab7-465a-937f-f19fe48b82b0" containerName="cleanup" Nov 01 23:00:36.857648 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857507 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a79bda1-5574-4dd9-9751-06804463a700" containerName="cleanup" Nov 01 23:00:36.857648 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857516 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="206fbb7b-ceea-4b88-98b9-5359edccfc01" containerName="cleanup" Nov 01 23:00:36.857648 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857522 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="78077924-93c1-4a66-a79a-bd391195e3ce" containerName="cleanup" Nov 01 23:00:36.857648 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857529 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="78077924-93c1-4a66-a79a-bd391195e3ce" containerName="cleanup" Nov 01 23:00:36.857648 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857535 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb7867dc-31eb-4d53-acb5-feabbdbea02a" containerName="cleanup" Nov 01 23:00:36.857648 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857546 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb9f7e76-0ab7-465a-937f-f19fe48b82b0" containerName="cleanup" Nov 01 23:00:36.857648 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857552 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb9f7e76-0ab7-465a-937f-f19fe48b82b0" containerName="cleanup" Nov 01 23:00:36.857648 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857557 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb7867dc-31eb-4d53-acb5-feabbdbea02a" containerName="cleanup" Nov 01 23:00:36.857648 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857562 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="206fbb7b-ceea-4b88-98b9-5359edccfc01" containerName="cleanup" Nov 01 23:00:36.857648 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857569 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb9f7e76-0ab7-465a-937f-f19fe48b82b0" containerName="cleanup" Nov 01 23:00:36.857648 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.857576 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a79bda1-5574-4dd9-9751-06804463a700" containerName="cleanup" Nov 01 23:00:36.863237 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.863217 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:36.870009 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.869970 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-7366bf-openshift-default-dockercfg-9nnps\"" Nov 01 23:00:36.874756 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:36.874733 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf"] Nov 01 23:00:37.056650 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:37.056606 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-workload-certs\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:37.056880 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:37.056660 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istio-envoy\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:37.056880 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:37.056812 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istio-podinfo\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:37.056880 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:37.056857 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istio-data\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:37.057065 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:37.056943 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-credential-socket\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:37.057065 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:37.056992 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q6v6\" (UniqueName: \"kubernetes.io/projected/efff7825-e6d6-4049-a3f2-620a4aeab8a2-kube-api-access-5q6v6\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:37.057175 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:37.057100 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istio-token\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:37.057175 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:37.057123 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istiod-ca-cert\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:37.057274 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:37.057241 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-workload-socket\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:37.159231 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:37.159140 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-workload-certs\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:37.159231 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:37.159198 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istio-envoy\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:37.159724 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:37.159340 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istio-podinfo\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:37.159724 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:37.159382 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istio-data\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:37.159724 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:37.159435 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-credential-socket\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:37.159724 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:37.159482 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q6v6\" (UniqueName: \"kubernetes.io/projected/efff7825-e6d6-4049-a3f2-620a4aeab8a2-kube-api-access-5q6v6\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:37.159724 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:37.159548 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-workload-certs\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:37.159724 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:37.159588 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istio-token\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:37.159724 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:37.159621 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istiod-ca-cert\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:37.160388 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:37.159745 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istio-data\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:37.160388 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:37.159764 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-workload-socket\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:37.160388 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:37.159789 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-credential-socket\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:37.160388 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:37.160307 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-workload-socket\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:37.160616 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:37.160522 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istiod-ca-cert\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:37.161643 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:37.161618 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istio-envoy\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:37.162097 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:37.162079 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istio-podinfo\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:37.168284 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:37.168261 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q6v6\" (UniqueName: \"kubernetes.io/projected/efff7825-e6d6-4049-a3f2-620a4aeab8a2-kube-api-access-5q6v6\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:37.168378 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:37.168280 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istio-token\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:37.182349 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:37.182322 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:37.477986 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:00:37.477950 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefff7825_e6d6_4049_a3f2_620a4aeab8a2.slice/crio-00c2df41fe7c6a8c64e8ae0c1e0f0a4d33202f79ecd462839c27c628790b22a7 WatchSource:0}: Error finding container 00c2df41fe7c6a8c64e8ae0c1e0f0a4d33202f79ecd462839c27c628790b22a7: Status 404 returned error can't find the container with id 00c2df41fe7c6a8c64e8ae0c1e0f0a4d33202f79ecd462839c27c628790b22a7 Nov 01 23:00:37.478324 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:37.478303 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf"] Nov 01 23:00:37.480995 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:37.480960 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:00:37.481084 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:37.481027 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:00:37.481084 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:37.481058 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:00:38.115256 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:38.115206 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" event={"ID":"efff7825-e6d6-4049-a3f2-620a4aeab8a2","Type":"ContainerStarted","Data":"a8f369b2610076be7250803dcc77123f081a6b0aa2ff9aba8d90509756e2cdf9"} Nov 01 23:00:38.115871 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:38.115534 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" event={"ID":"efff7825-e6d6-4049-a3f2-620a4aeab8a2","Type":"ContainerStarted","Data":"00c2df41fe7c6a8c64e8ae0c1e0f0a4d33202f79ecd462839c27c628790b22a7"} Nov 01 23:00:38.140716 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:38.140663 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" podStartSLOduration=2.140647355 podStartE2EDuration="2.140647355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:38.136905155 +0000 UTC m=+2206.541495787" watchObservedRunningTime="2026-11-01 23:00:38.140647355 +0000 UTC m=+2206.545237987" Nov 01 23:00:38.182762 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:38.182735 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:38.184433 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:38.184405 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 01 23:00:38.184553 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:38.184454 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" podUID="efff7825-e6d6-4049-a3f2-620a4aeab8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 01 23:00:39.183334 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:39.183289 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 01 23:00:39.183728 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:39.183347 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" podUID="efff7825-e6d6-4049-a3f2-620a4aeab8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 01 23:00:40.183011 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:40.182968 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 01 23:00:40.183240 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:40.183033 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" podUID="efff7825-e6d6-4049-a3f2-620a4aeab8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 01 23:00:41.183474 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:41.183431 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 01 23:00:41.183943 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:41.183505 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" podUID="efff7825-e6d6-4049-a3f2-620a4aeab8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 01 23:00:41.730857 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:41.728712 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f"] Nov 01 23:00:41.907862 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:41.901288 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" podUID="665520e7-d3f3-4dd9-a7a5-3992fd827544" containerName="istio-proxy" containerID="cri-o://cbd00e2dc954916bb266d48e01ba6febbf7539e77742536a71c145f8147aa0d9" gracePeriod=30 Nov 01 23:00:42.183405 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:42.183369 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 01 23:00:42.183593 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:42.183425 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" podUID="efff7825-e6d6-4049-a3f2-620a4aeab8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 01 23:00:43.183374 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:43.183339 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 01 23:00:43.183554 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:43.183399 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" podUID="efff7825-e6d6-4049-a3f2-620a4aeab8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 01 23:00:44.159720 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:44.159690 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-021714-7b74898bd7-s2gqb" Nov 01 23:00:44.184885 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:44.184344 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 01 23:00:44.184885 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:44.184419 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" podUID="efff7825-e6d6-4049-a3f2-620a4aeab8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 01 23:00:44.205215 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:44.205177 5011 generic.go:356] "Generic (PLEG): container finished" podID="02975ed3-18c1-49ee-8bae-e86edb4e0130" containerID="fe589a25553a50e5ab341d7a39a63745bab2a6c58e2d8ea925ffad304e292bcf" exitCode=0 Nov 01 23:00:44.205450 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:44.205254 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-021714-7b74898bd7-s2gqb" Nov 01 23:00:44.205450 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:44.205335 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-021714-7b74898bd7-s2gqb" event={"ID":"02975ed3-18c1-49ee-8bae-e86edb4e0130","Type":"ContainerDied","Data":"fe589a25553a50e5ab341d7a39a63745bab2a6c58e2d8ea925ffad304e292bcf"} Nov 01 23:00:44.205450 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:44.205388 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-021714-7b74898bd7-s2gqb" event={"ID":"02975ed3-18c1-49ee-8bae-e86edb4e0130","Type":"ContainerDied","Data":"c2f48fa1c1327f802a24a8327e0d401ade358c66f857d9936997d29124c434df"} Nov 01 23:00:44.205450 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:44.205418 5011 scope.go:122] "RemoveContainer" containerID="fe589a25553a50e5ab341d7a39a63745bab2a6c58e2d8ea925ffad304e292bcf" Nov 01 23:00:44.224654 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:44.224627 5011 scope.go:122] "RemoveContainer" containerID="fe589a25553a50e5ab341d7a39a63745bab2a6c58e2d8ea925ffad304e292bcf" Nov 01 23:00:44.225108 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:00:44.225067 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe589a25553a50e5ab341d7a39a63745bab2a6c58e2d8ea925ffad304e292bcf\": container with ID starting with fe589a25553a50e5ab341d7a39a63745bab2a6c58e2d8ea925ffad304e292bcf not found: ID does not exist" containerID="fe589a25553a50e5ab341d7a39a63745bab2a6c58e2d8ea925ffad304e292bcf" Nov 01 23:00:44.225237 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:44.225115 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe589a25553a50e5ab341d7a39a63745bab2a6c58e2d8ea925ffad304e292bcf"} err="failed to get container status \"fe589a25553a50e5ab341d7a39a63745bab2a6c58e2d8ea925ffad304e292bcf\": rpc error: code = NotFound desc = could not find container \"fe589a25553a50e5ab341d7a39a63745bab2a6c58e2d8ea925ffad304e292bcf\": container with ID starting with fe589a25553a50e5ab341d7a39a63745bab2a6c58e2d8ea925ffad304e292bcf not found: ID does not exist" Nov 01 23:00:44.326453 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:44.326404 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02975ed3-18c1-49ee-8bae-e86edb4e0130-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02975ed3-18c1-49ee-8bae-e86edb4e0130-plugins-config-volume\") pod \"02975ed3-18c1-49ee-8bae-e86edb4e0130\" (UID: \"02975ed3-18c1-49ee-8bae-e86edb4e0130\") " Nov 01 23:00:44.326453 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:44.326459 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02975ed3-18c1-49ee-8bae-e86edb4e0130-kube-api-access-2m42g\" (UniqueName: \"kubernetes.io/projected/02975ed3-18c1-49ee-8bae-e86edb4e0130-kube-api-access-2m42g\") pod \"02975ed3-18c1-49ee-8bae-e86edb4e0130\" (UID: \"02975ed3-18c1-49ee-8bae-e86edb4e0130\") " Nov 01 23:00:44.326971 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:44.326942 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02975ed3-18c1-49ee-8bae-e86edb4e0130-plugins-config-volume" pod "02975ed3-18c1-49ee-8bae-e86edb4e0130" (UID: "02975ed3-18c1-49ee-8bae-e86edb4e0130"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:44.327515 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:44.327491 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02975ed3-18c1-49ee-8bae-e86edb4e0130-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:44.328990 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:44.328965 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02975ed3-18c1-49ee-8bae-e86edb4e0130-kube-api-access-2m42g" pod "02975ed3-18c1-49ee-8bae-e86edb4e0130" (UID: "02975ed3-18c1-49ee-8bae-e86edb4e0130"). InnerVolumeSpecName "kube-api-access-2m42g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:44.429305 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:44.429261 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m42g\" (UniqueName: \"kubernetes.io/projected/02975ed3-18c1-49ee-8bae-e86edb4e0130-kube-api-access-2m42g\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:44.537568 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:44.537534 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-021714-7b74898bd7-s2gqb"] Nov 01 23:00:44.544346 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:44.544307 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-021714-7b74898bd7-s2gqb"] Nov 01 23:00:44.544560 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:00:44.544539 5011 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02975ed3_18c1_49ee_8bae_e86edb4e0130.slice\": RecentStats: unable to find data in memory cache]" Nov 01 23:00:45.188224 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:45.188190 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:45.188686 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:45.188672 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:45.189505 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:45.189481 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:00:45.772011 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:45.771978 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02975ed3-18c1-49ee-8bae-e86edb4e0130" path="/var/lib/kubelet/pods/02975ed3-18c1-49ee-8bae-e86edb4e0130/volumes" Nov 01 23:00:46.153681 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.153590 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2"] Nov 01 23:00:46.153897 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.153858 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2" podUID="da68866f-9cf3-4467-b876-6ac92fcf6c3c" containerName="maas-api" containerID="cri-o://acc33b8c62bc85baad83ad6324df1002c9ee26c9a7e80b0268d31ca81be3886b" gracePeriod=30 Nov 01 23:00:46.689261 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.689227 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2" Nov 01 23:00:46.693256 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.693214 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm"] Nov 01 23:00:46.694314 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.694292 5011 cpu_manager.go:414] "containerMap: removing container" podUID="da68866f-9cf3-4467-b876-6ac92fcf6c3c" containerName="maas-api" Nov 01 23:00:46.694314 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.694313 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da68866f-9cf3-4467-b876-6ac92fcf6c3c" containerName="maas-api" Nov 01 23:00:46.694462 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.694332 5011 cpu_manager.go:414] "containerMap: removing container" podUID="206fbb7b-ceea-4b88-98b9-5359edccfc01" containerName="cleanup" Nov 01 23:00:46.694462 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.694338 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="206fbb7b-ceea-4b88-98b9-5359edccfc01" containerName="cleanup" Nov 01 23:00:46.694462 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.694351 5011 cpu_manager.go:414] "containerMap: removing container" podUID="9a79bda1-5574-4dd9-9751-06804463a700" containerName="cleanup" Nov 01 23:00:46.694462 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.694355 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a79bda1-5574-4dd9-9751-06804463a700" containerName="cleanup" Nov 01 23:00:46.694462 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.694363 5011 cpu_manager.go:414] "containerMap: removing container" podUID="fb9f7e76-0ab7-465a-937f-f19fe48b82b0" containerName="cleanup" Nov 01 23:00:46.694462 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.694367 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb9f7e76-0ab7-465a-937f-f19fe48b82b0" containerName="cleanup" Nov 01 23:00:46.694462 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.694390 5011 cpu_manager.go:414] "containerMap: removing container" podUID="02975ed3-18c1-49ee-8bae-e86edb4e0130" containerName="payload-processing" Nov 01 23:00:46.694462 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.694394 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02975ed3-18c1-49ee-8bae-e86edb4e0130" containerName="payload-processing" Nov 01 23:00:46.694718 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.694533 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="206fbb7b-ceea-4b88-98b9-5359edccfc01" containerName="cleanup" Nov 01 23:00:46.694718 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.694550 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="02975ed3-18c1-49ee-8bae-e86edb4e0130" containerName="payload-processing" Nov 01 23:00:46.694718 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.694558 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a79bda1-5574-4dd9-9751-06804463a700" containerName="cleanup" Nov 01 23:00:46.694718 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.694565 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="78077924-93c1-4a66-a79a-bd391195e3ce" containerName="cleanup" Nov 01 23:00:46.694718 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.694573 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb7867dc-31eb-4d53-acb5-feabbdbea02a" containerName="cleanup" Nov 01 23:00:46.694718 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.694592 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="da68866f-9cf3-4467-b876-6ac92fcf6c3c" containerName="maas-api" Nov 01 23:00:46.698299 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.698269 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:46.707581 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.707559 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-075771-openshift-default-dockercfg-ht2bd\"" Nov 01 23:00:46.714815 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.714786 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm"] Nov 01 23:00:46.790605 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.790557 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da68866f-9cf3-4467-b876-6ac92fcf6c3c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da68866f-9cf3-4467-b876-6ac92fcf6c3c-maas-api-tls\") pod \"da68866f-9cf3-4467-b876-6ac92fcf6c3c\" (UID: \"da68866f-9cf3-4467-b876-6ac92fcf6c3c\") " Nov 01 23:00:46.790779 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.790635 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da68866f-9cf3-4467-b876-6ac92fcf6c3c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da68866f-9cf3-4467-b876-6ac92fcf6c3c-metrics-certs\") pod \"da68866f-9cf3-4467-b876-6ac92fcf6c3c\" (UID: \"da68866f-9cf3-4467-b876-6ac92fcf6c3c\") " Nov 01 23:00:46.790871 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.790779 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da68866f-9cf3-4467-b876-6ac92fcf6c3c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da68866f-9cf3-4467-b876-6ac92fcf6c3c-openshift-service-ca\") pod \"da68866f-9cf3-4467-b876-6ac92fcf6c3c\" (UID: \"da68866f-9cf3-4467-b876-6ac92fcf6c3c\") " Nov 01 23:00:46.791060 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.791035 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da68866f-9cf3-4467-b876-6ac92fcf6c3c-kube-api-access-jrzvp\" (UniqueName: \"kubernetes.io/projected/da68866f-9cf3-4467-b876-6ac92fcf6c3c-kube-api-access-jrzvp\") pod \"da68866f-9cf3-4467-b876-6ac92fcf6c3c\" (UID: \"da68866f-9cf3-4467-b876-6ac92fcf6c3c\") " Nov 01 23:00:46.791165 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.791138 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da68866f-9cf3-4467-b876-6ac92fcf6c3c-openshift-service-ca" pod "da68866f-9cf3-4467-b876-6ac92fcf6c3c" (UID: "da68866f-9cf3-4467-b876-6ac92fcf6c3c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:46.791543 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.791517 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-workload-socket\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:46.791914 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.791890 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-credential-socket\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:46.792040 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.791936 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-workload-certs\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:46.792126 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.792105 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq9bc\" (UniqueName: \"kubernetes.io/projected/e6f28eed-cd8b-472c-ae3a-75722591e360-kube-api-access-nq9bc\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:46.792300 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.792277 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-istio-data\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:46.792369 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.792346 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-istio-envoy\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:46.792441 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.792399 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6f28eed-cd8b-472c-ae3a-75722591e360-istio-token\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:46.792558 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.792540 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6f28eed-cd8b-472c-ae3a-75722591e360-istiod-ca-cert\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:46.792726 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.792707 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6f28eed-cd8b-472c-ae3a-75722591e360-istio-podinfo\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:46.792939 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.792921 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da68866f-9cf3-4467-b876-6ac92fcf6c3c-openshift-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:46.793198 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.793173 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da68866f-9cf3-4467-b876-6ac92fcf6c3c-maas-api-tls" pod "da68866f-9cf3-4467-b876-6ac92fcf6c3c" (UID: "da68866f-9cf3-4467-b876-6ac92fcf6c3c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:00:46.793334 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.793316 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da68866f-9cf3-4467-b876-6ac92fcf6c3c-kube-api-access-jrzvp" pod "da68866f-9cf3-4467-b876-6ac92fcf6c3c" (UID: "da68866f-9cf3-4467-b876-6ac92fcf6c3c"). InnerVolumeSpecName "kube-api-access-jrzvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:46.793948 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.793925 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da68866f-9cf3-4467-b876-6ac92fcf6c3c-metrics-certs" pod "da68866f-9cf3-4467-b876-6ac92fcf6c3c" (UID: "da68866f-9cf3-4467-b876-6ac92fcf6c3c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:00:46.907907 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.900980 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6f28eed-cd8b-472c-ae3a-75722591e360-istio-podinfo\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:46.907907 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.901098 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-workload-socket\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:46.907907 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.901217 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-credential-socket\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:46.907907 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.901247 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-workload-certs\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:46.907907 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.901352 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq9bc\" (UniqueName: \"kubernetes.io/projected/e6f28eed-cd8b-472c-ae3a-75722591e360-kube-api-access-nq9bc\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:46.907907 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.901452 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-istio-data\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:46.907907 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.901500 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-istio-envoy\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:46.907907 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.901524 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6f28eed-cd8b-472c-ae3a-75722591e360-istio-token\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:46.907907 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.901597 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6f28eed-cd8b-472c-ae3a-75722591e360-istiod-ca-cert\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:46.907907 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.901705 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrzvp\" (UniqueName: \"kubernetes.io/projected/da68866f-9cf3-4467-b876-6ac92fcf6c3c-kube-api-access-jrzvp\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:46.907907 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.901722 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da68866f-9cf3-4467-b876-6ac92fcf6c3c-maas-api-tls\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:46.907907 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.901736 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da68866f-9cf3-4467-b876-6ac92fcf6c3c-metrics-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:46.907907 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.902481 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6f28eed-cd8b-472c-ae3a-75722591e360-istiod-ca-cert\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:46.919907 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.911798 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-workload-certs\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:46.919907 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.912582 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-credential-socket\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:46.919907 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.913229 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-istio-data\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:46.919907 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.915224 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq9bc\" (UniqueName: \"kubernetes.io/projected/e6f28eed-cd8b-472c-ae3a-75722591e360-kube-api-access-nq9bc\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:46.919907 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.915612 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-workload-socket\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:46.921680 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.921622 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6f28eed-cd8b-472c-ae3a-75722591e360-istio-podinfo\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:46.927082 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.922478 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6f28eed-cd8b-472c-ae3a-75722591e360-istio-token\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:46.929933 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:46.927359 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-istio-envoy\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:46.986430 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:00:46.986387 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:00:47.016464 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.016430 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:47.271384 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.271349 5011 generic.go:356] "Generic (PLEG): container finished" podID="da68866f-9cf3-4467-b876-6ac92fcf6c3c" containerID="acc33b8c62bc85baad83ad6324df1002c9ee26c9a7e80b0268d31ca81be3886b" exitCode=0 Nov 01 23:00:47.271540 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.271403 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2" event={"ID":"da68866f-9cf3-4467-b876-6ac92fcf6c3c","Type":"ContainerDied","Data":"acc33b8c62bc85baad83ad6324df1002c9ee26c9a7e80b0268d31ca81be3886b"} Nov 01 23:00:47.271540 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.271434 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2" Nov 01 23:00:47.271540 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.271472 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2" event={"ID":"da68866f-9cf3-4467-b876-6ac92fcf6c3c","Type":"ContainerDied","Data":"a4ca22e801c151bc00e75375e950afdeee3c072a4db716e047392389cd6f3f25"} Nov 01 23:00:47.271540 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.271504 5011 scope.go:122] "RemoveContainer" containerID="acc33b8c62bc85baad83ad6324df1002c9ee26c9a7e80b0268d31ca81be3886b" Nov 01 23:00:47.282878 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.277656 5011 generic.go:356] "Generic (PLEG): container finished" podID="665520e7-d3f3-4dd9-a7a5-3992fd827544" containerID="cbd00e2dc954916bb266d48e01ba6febbf7539e77742536a71c145f8147aa0d9" exitCode=0 Nov 01 23:00:47.282878 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.277806 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" event={"ID":"665520e7-d3f3-4dd9-a7a5-3992fd827544","Type":"ContainerDied","Data":"cbd00e2dc954916bb266d48e01ba6febbf7539e77742536a71c145f8147aa0d9"} Nov 01 23:00:47.315165 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.315108 5011 scope.go:122] "RemoveContainer" containerID="acc33b8c62bc85baad83ad6324df1002c9ee26c9a7e80b0268d31ca81be3886b" Nov 01 23:00:47.321455 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:00:47.321310 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acc33b8c62bc85baad83ad6324df1002c9ee26c9a7e80b0268d31ca81be3886b\": container with ID starting with acc33b8c62bc85baad83ad6324df1002c9ee26c9a7e80b0268d31ca81be3886b not found: ID does not exist" containerID="acc33b8c62bc85baad83ad6324df1002c9ee26c9a7e80b0268d31ca81be3886b" Nov 01 23:00:47.321455 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.321360 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acc33b8c62bc85baad83ad6324df1002c9ee26c9a7e80b0268d31ca81be3886b"} err="failed to get container status \"acc33b8c62bc85baad83ad6324df1002c9ee26c9a7e80b0268d31ca81be3886b\": rpc error: code = NotFound desc = could not find container \"acc33b8c62bc85baad83ad6324df1002c9ee26c9a7e80b0268d31ca81be3886b\": container with ID starting with acc33b8c62bc85baad83ad6324df1002c9ee26c9a7e80b0268d31ca81be3886b not found: ID does not exist" Nov 01 23:00:47.321455 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.321387 5011 scope.go:122] "RemoveContainer" containerID="dd45e515bc228815d2ab2ea472d08728c59cdb6b2bb7dd947fb3e9ac0a2414bd" Nov 01 23:00:47.321771 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.321745 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 23:00:47.322311 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.322278 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm"] Nov 01 23:00:47.328664 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.328623 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:00:47.328780 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.328712 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:00:47.328780 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.328756 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:00:47.328940 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.328914 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2"] Nov 01 23:00:47.333027 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.333002 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2f675d-68f899cf58-bngh2"] Nov 01 23:00:47.441339 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.441304 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/665520e7-d3f3-4dd9-a7a5-3992fd827544-istio-token\" (UniqueName: \"kubernetes.io/projected/665520e7-d3f3-4dd9-a7a5-3992fd827544-istio-token\") pod \"665520e7-d3f3-4dd9-a7a5-3992fd827544\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " Nov 01 23:00:47.441527 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.441367 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/665520e7-d3f3-4dd9-a7a5-3992fd827544-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/665520e7-d3f3-4dd9-a7a5-3992fd827544-istio-podinfo\") pod \"665520e7-d3f3-4dd9-a7a5-3992fd827544\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " Nov 01 23:00:47.441527 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.441502 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/665520e7-d3f3-4dd9-a7a5-3992fd827544-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/665520e7-d3f3-4dd9-a7a5-3992fd827544-istiod-ca-cert\") pod \"665520e7-d3f3-4dd9-a7a5-3992fd827544\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " Nov 01 23:00:47.441667 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.441564 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-workload-certs\") pod \"665520e7-d3f3-4dd9-a7a5-3992fd827544\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " Nov 01 23:00:47.441667 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.441598 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-istio-envoy\") pod \"665520e7-d3f3-4dd9-a7a5-3992fd827544\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " Nov 01 23:00:47.441783 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.441670 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/665520e7-d3f3-4dd9-a7a5-3992fd827544-kube-api-access-glgwm\" (UniqueName: \"kubernetes.io/projected/665520e7-d3f3-4dd9-a7a5-3992fd827544-kube-api-access-glgwm\") pod \"665520e7-d3f3-4dd9-a7a5-3992fd827544\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " Nov 01 23:00:47.441783 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.441729 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-istio-data\") pod \"665520e7-d3f3-4dd9-a7a5-3992fd827544\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " Nov 01 23:00:47.441954 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.441827 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-credential-socket\") pod \"665520e7-d3f3-4dd9-a7a5-3992fd827544\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " Nov 01 23:00:47.441954 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.441878 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-workload-socket\") pod \"665520e7-d3f3-4dd9-a7a5-3992fd827544\" (UID: \"665520e7-d3f3-4dd9-a7a5-3992fd827544\") " Nov 01 23:00:47.441954 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.441915 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/665520e7-d3f3-4dd9-a7a5-3992fd827544-istiod-ca-cert" pod "665520e7-d3f3-4dd9-a7a5-3992fd827544" (UID: "665520e7-d3f3-4dd9-a7a5-3992fd827544"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:47.442160 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.442135 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-workload-certs" pod "665520e7-d3f3-4dd9-a7a5-3992fd827544" (UID: "665520e7-d3f3-4dd9-a7a5-3992fd827544"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.442160 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.442150 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-istio-data" pod "665520e7-d3f3-4dd9-a7a5-3992fd827544" (UID: "665520e7-d3f3-4dd9-a7a5-3992fd827544"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.442775 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.442663 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-workload-socket" pod "665520e7-d3f3-4dd9-a7a5-3992fd827544" (UID: "665520e7-d3f3-4dd9-a7a5-3992fd827544"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.443238 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.443211 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.443346 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.443244 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.443346 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.443261 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/665520e7-d3f3-4dd9-a7a5-3992fd827544-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.443346 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.443276 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.443510 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.443380 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-credential-socket" pod "665520e7-d3f3-4dd9-a7a5-3992fd827544" (UID: "665520e7-d3f3-4dd9-a7a5-3992fd827544"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.443950 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.443928 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/665520e7-d3f3-4dd9-a7a5-3992fd827544-istio-podinfo" pod "665520e7-d3f3-4dd9-a7a5-3992fd827544" (UID: "665520e7-d3f3-4dd9-a7a5-3992fd827544"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:00:47.444160 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.444116 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-istio-envoy" pod "665520e7-d3f3-4dd9-a7a5-3992fd827544" (UID: "665520e7-d3f3-4dd9-a7a5-3992fd827544"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.444424 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.444405 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/665520e7-d3f3-4dd9-a7a5-3992fd827544-kube-api-access-glgwm" pod "665520e7-d3f3-4dd9-a7a5-3992fd827544" (UID: "665520e7-d3f3-4dd9-a7a5-3992fd827544"). InnerVolumeSpecName "kube-api-access-glgwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:47.444485 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.444419 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/665520e7-d3f3-4dd9-a7a5-3992fd827544-istio-token" pod "665520e7-d3f3-4dd9-a7a5-3992fd827544" (UID: "665520e7-d3f3-4dd9-a7a5-3992fd827544"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:47.545186 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.545127 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/665520e7-d3f3-4dd9-a7a5-3992fd827544-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.545186 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.545176 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/665520e7-d3f3-4dd9-a7a5-3992fd827544-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.545186 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.545190 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.545450 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.545203 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glgwm\" (UniqueName: \"kubernetes.io/projected/665520e7-d3f3-4dd9-a7a5-3992fd827544-kube-api-access-glgwm\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.545450 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.545219 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/665520e7-d3f3-4dd9-a7a5-3992fd827544-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.810764 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:47.810678 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da68866f-9cf3-4467-b876-6ac92fcf6c3c" path="/var/lib/kubelet/pods/da68866f-9cf3-4467-b876-6ac92fcf6c3c/volumes" Nov 01 23:00:48.287687 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:48.287644 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" event={"ID":"e6f28eed-cd8b-472c-ae3a-75722591e360","Type":"ContainerStarted","Data":"ee6bc720f86ac7ba0ba89d485f08aeaab08b91d489e9f4791cada0175eeaadb5"} Nov 01 23:00:48.287890 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:48.287708 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" event={"ID":"e6f28eed-cd8b-472c-ae3a-75722591e360","Type":"ContainerStarted","Data":"1cdec6ffb1357cf27554635cb106067700b7f47fe90921bc2701d50cab2b6b3e"} Nov 01 23:00:48.290199 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:48.290178 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" Nov 01 23:00:48.290199 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:48.290191 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f" event={"ID":"665520e7-d3f3-4dd9-a7a5-3992fd827544","Type":"ContainerDied","Data":"101415afcfcc565e264a83897c47659a472836fcd379f8b586ab08d1206e3601"} Nov 01 23:00:48.290358 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:48.290243 5011 scope.go:122] "RemoveContainer" containerID="cbd00e2dc954916bb266d48e01ba6febbf7539e77742536a71c145f8147aa0d9" Nov 01 23:00:48.313046 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:48.312998 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" podStartSLOduration=2.31298453 podStartE2EDuration="2.31298453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:48.308360575 +0000 UTC m=+2216.712951230" watchObservedRunningTime="2026-11-01 23:00:48.31298453 +0000 UTC m=+2216.717575161" Nov 01 23:00:48.353092 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:48.353051 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f"] Nov 01 23:00:48.358417 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:48.358382 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-52r2f"] Nov 01 23:00:49.016571 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:49.016537 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:49.017743 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:49.017721 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Nov 01 23:00:49.017899 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:49.017770 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" podUID="e6f28eed-cd8b-472c-ae3a-75722591e360" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Nov 01 23:00:49.777785 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:49.777753 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="665520e7-d3f3-4dd9-a7a5-3992fd827544" path="/var/lib/kubelet/pods/665520e7-d3f3-4dd9-a7a5-3992fd827544/volumes" Nov 01 23:00:50.018684 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:50.018645 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Nov 01 23:00:50.019093 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:50.018722 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" podUID="e6f28eed-cd8b-472c-ae3a-75722591e360" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Nov 01 23:00:51.035856 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:51.035426 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Nov 01 23:00:51.035856 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:51.035493 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" podUID="e6f28eed-cd8b-472c-ae3a-75722591e360" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Nov 01 23:00:52.016889 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:52.016814 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Nov 01 23:00:52.017095 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:52.016898 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" podUID="e6f28eed-cd8b-472c-ae3a-75722591e360" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Nov 01 23:00:53.017860 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:53.017683 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Nov 01 23:00:53.017860 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:53.017754 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" podUID="e6f28eed-cd8b-472c-ae3a-75722591e360" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Nov 01 23:00:53.439261 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:53.439230 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh"] Nov 01 23:00:53.439502 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:53.439465 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" podUID="deb96044-ee0a-4050-8076-ebce2c1d990a" containerName="istio-proxy" containerID="cri-o://8f5ae1bd5a2431219ec22ebc3f97d5b7c8d871b2f6278cf73bbba06c71ca1679" gracePeriod=30 Nov 01 23:00:54.028561 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:54.028414 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Nov 01 23:00:54.028561 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:54.028485 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" podUID="e6f28eed-cd8b-472c-ae3a-75722591e360" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Nov 01 23:00:55.068747 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:55.068127 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:55.071861 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:55.069680 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:55.072822 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:55.072372 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:00:58.930360 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:58.930293 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 23:00:58.960273 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:58.960242 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deb96044-ee0a-4050-8076-ebce2c1d990a-istio-token\" (UniqueName: \"kubernetes.io/projected/deb96044-ee0a-4050-8076-ebce2c1d990a-istio-token\") pod \"deb96044-ee0a-4050-8076-ebce2c1d990a\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " Nov 01 23:00:58.960442 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:58.960295 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-istio-envoy\") pod \"deb96044-ee0a-4050-8076-ebce2c1d990a\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " Nov 01 23:00:58.960442 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:58.960328 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deb96044-ee0a-4050-8076-ebce2c1d990a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb96044-ee0a-4050-8076-ebce2c1d990a-istiod-ca-cert\") pod \"deb96044-ee0a-4050-8076-ebce2c1d990a\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " Nov 01 23:00:58.960442 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:58.960345 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/deb96044-ee0a-4050-8076-ebce2c1d990a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb96044-ee0a-4050-8076-ebce2c1d990a-istio-podinfo\") pod \"deb96044-ee0a-4050-8076-ebce2c1d990a\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " Nov 01 23:00:58.960442 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:58.960363 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deb96044-ee0a-4050-8076-ebce2c1d990a-kube-api-access-r4bm7\" (UniqueName: \"kubernetes.io/projected/deb96044-ee0a-4050-8076-ebce2c1d990a-kube-api-access-r4bm7\") pod \"deb96044-ee0a-4050-8076-ebce2c1d990a\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " Nov 01 23:00:58.960442 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:58.960395 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-workload-socket\") pod \"deb96044-ee0a-4050-8076-ebce2c1d990a\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " Nov 01 23:00:58.960729 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:58.960650 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-workload-certs\") pod \"deb96044-ee0a-4050-8076-ebce2c1d990a\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " Nov 01 23:00:58.960729 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:58.960688 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-credential-socket\") pod \"deb96044-ee0a-4050-8076-ebce2c1d990a\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " Nov 01 23:00:58.960729 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:58.960723 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-istio-data\") pod \"deb96044-ee0a-4050-8076-ebce2c1d990a\" (UID: \"deb96044-ee0a-4050-8076-ebce2c1d990a\") " Nov 01 23:00:58.960984 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:58.960927 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deb96044-ee0a-4050-8076-ebce2c1d990a-istiod-ca-cert" pod "deb96044-ee0a-4050-8076-ebce2c1d990a" (UID: "deb96044-ee0a-4050-8076-ebce2c1d990a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:58.962153 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:58.961045 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-workload-certs" pod "deb96044-ee0a-4050-8076-ebce2c1d990a" (UID: "deb96044-ee0a-4050-8076-ebce2c1d990a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:58.962153 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:58.961123 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-workload-socket" pod "deb96044-ee0a-4050-8076-ebce2c1d990a" (UID: "deb96044-ee0a-4050-8076-ebce2c1d990a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:58.962153 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:58.961139 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-credential-socket" pod "deb96044-ee0a-4050-8076-ebce2c1d990a" (UID: "deb96044-ee0a-4050-8076-ebce2c1d990a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:58.962153 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:58.961683 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:58.962153 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:58.961706 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:58.962153 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:58.961721 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb96044-ee0a-4050-8076-ebce2c1d990a-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:58.962153 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:58.961735 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:58.962153 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:58.962020 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-istio-data" pod "deb96044-ee0a-4050-8076-ebce2c1d990a" (UID: "deb96044-ee0a-4050-8076-ebce2c1d990a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:58.963085 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:58.963056 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/deb96044-ee0a-4050-8076-ebce2c1d990a-istio-podinfo" pod "deb96044-ee0a-4050-8076-ebce2c1d990a" (UID: "deb96044-ee0a-4050-8076-ebce2c1d990a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:00:58.963467 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:58.963448 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-istio-envoy" pod "deb96044-ee0a-4050-8076-ebce2c1d990a" (UID: "deb96044-ee0a-4050-8076-ebce2c1d990a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:58.963548 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:58.963535 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb96044-ee0a-4050-8076-ebce2c1d990a-istio-token" pod "deb96044-ee0a-4050-8076-ebce2c1d990a" (UID: "deb96044-ee0a-4050-8076-ebce2c1d990a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:58.964056 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:58.964032 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb96044-ee0a-4050-8076-ebce2c1d990a-kube-api-access-r4bm7" pod "deb96044-ee0a-4050-8076-ebce2c1d990a" (UID: "deb96044-ee0a-4050-8076-ebce2c1d990a"). InnerVolumeSpecName "kube-api-access-r4bm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:59.063009 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:59.062970 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:59.063009 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:59.063003 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb96044-ee0a-4050-8076-ebce2c1d990a-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:59.063194 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:59.063017 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb96044-ee0a-4050-8076-ebce2c1d990a-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:59.063194 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:59.063030 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb96044-ee0a-4050-8076-ebce2c1d990a-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:59.063194 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:59.063042 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4bm7\" (UniqueName: \"kubernetes.io/projected/deb96044-ee0a-4050-8076-ebce2c1d990a-kube-api-access-r4bm7\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:59.493610 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:59.493572 5011 generic.go:356] "Generic (PLEG): container finished" podID="deb96044-ee0a-4050-8076-ebce2c1d990a" containerID="8f5ae1bd5a2431219ec22ebc3f97d5b7c8d871b2f6278cf73bbba06c71ca1679" exitCode=0 Nov 01 23:00:59.493789 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:59.493630 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" Nov 01 23:00:59.493789 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:59.493655 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" event={"ID":"deb96044-ee0a-4050-8076-ebce2c1d990a","Type":"ContainerDied","Data":"8f5ae1bd5a2431219ec22ebc3f97d5b7c8d871b2f6278cf73bbba06c71ca1679"} Nov 01 23:00:59.493789 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:59.493713 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh" event={"ID":"deb96044-ee0a-4050-8076-ebce2c1d990a","Type":"ContainerDied","Data":"8175dee3f90f2bedbda87145dd11106e07cb23938464f1b732cce8310272c5b2"} Nov 01 23:00:59.493789 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:59.493735 5011 scope.go:122] "RemoveContainer" containerID="8f5ae1bd5a2431219ec22ebc3f97d5b7c8d871b2f6278cf73bbba06c71ca1679" Nov 01 23:00:59.514543 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:59.514507 5011 scope.go:122] "RemoveContainer" containerID="f4077be34fc1ad4c8666239c81e6092a00dba3cac0dd365251f732426419cc97" Nov 01 23:00:59.527328 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:59.527291 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh"] Nov 01 23:00:59.532327 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:59.532292 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2f675d-openshift-default-6db6cbf4bc-8l9hh"] Nov 01 23:00:59.535124 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:59.535097 5011 scope.go:122] "RemoveContainer" containerID="8f5ae1bd5a2431219ec22ebc3f97d5b7c8d871b2f6278cf73bbba06c71ca1679" Nov 01 23:00:59.535468 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:00:59.535443 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f5ae1bd5a2431219ec22ebc3f97d5b7c8d871b2f6278cf73bbba06c71ca1679\": container with ID starting with 8f5ae1bd5a2431219ec22ebc3f97d5b7c8d871b2f6278cf73bbba06c71ca1679 not found: ID does not exist" containerID="8f5ae1bd5a2431219ec22ebc3f97d5b7c8d871b2f6278cf73bbba06c71ca1679" Nov 01 23:00:59.535550 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:59.535480 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f5ae1bd5a2431219ec22ebc3f97d5b7c8d871b2f6278cf73bbba06c71ca1679"} err="failed to get container status \"8f5ae1bd5a2431219ec22ebc3f97d5b7c8d871b2f6278cf73bbba06c71ca1679\": rpc error: code = NotFound desc = could not find container \"8f5ae1bd5a2431219ec22ebc3f97d5b7c8d871b2f6278cf73bbba06c71ca1679\": container with ID starting with 8f5ae1bd5a2431219ec22ebc3f97d5b7c8d871b2f6278cf73bbba06c71ca1679 not found: ID does not exist" Nov 01 23:00:59.535550 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:59.535505 5011 scope.go:122] "RemoveContainer" containerID="f4077be34fc1ad4c8666239c81e6092a00dba3cac0dd365251f732426419cc97" Nov 01 23:00:59.535793 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:00:59.535775 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4077be34fc1ad4c8666239c81e6092a00dba3cac0dd365251f732426419cc97\": container with ID starting with f4077be34fc1ad4c8666239c81e6092a00dba3cac0dd365251f732426419cc97 not found: ID does not exist" containerID="f4077be34fc1ad4c8666239c81e6092a00dba3cac0dd365251f732426419cc97" Nov 01 23:00:59.535851 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:59.535799 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4077be34fc1ad4c8666239c81e6092a00dba3cac0dd365251f732426419cc97"} err="failed to get container status \"f4077be34fc1ad4c8666239c81e6092a00dba3cac0dd365251f732426419cc97\": rpc error: code = NotFound desc = could not find container \"f4077be34fc1ad4c8666239c81e6092a00dba3cac0dd365251f732426419cc97\": container with ID starting with f4077be34fc1ad4c8666239c81e6092a00dba3cac0dd365251f732426419cc97 not found: ID does not exist" Nov 01 23:00:59.749697 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:00:59.749621 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deb96044-ee0a-4050-8076-ebce2c1d990a" path="/var/lib/kubelet/pods/deb96044-ee0a-4050-8076-ebce2c1d990a/volumes" Nov 01 23:01:16.983213 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:01:16.983174 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:01:22.219180 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:22.219147 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:01:26.477863 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:26.477813 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs"] Nov 01 23:01:26.478682 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:26.478626 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" podUID="89a5b17f-1549-470c-8968-01b013df9417" containerName="istio-proxy" containerID="cri-o://a1466adfa6bed6f0484768245c91d5f57f158fd51a11e3bcc966c756dd195a39" gracePeriod=30 Nov 01 23:01:31.928909 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:31.928875 5011 generic.go:356] "Generic (PLEG): container finished" podID="89a5b17f-1549-470c-8968-01b013df9417" containerID="a1466adfa6bed6f0484768245c91d5f57f158fd51a11e3bcc966c756dd195a39" exitCode=0 Nov 01 23:01:31.929365 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:31.928927 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" event={"ID":"89a5b17f-1549-470c-8968-01b013df9417","Type":"ContainerDied","Data":"a1466adfa6bed6f0484768245c91d5f57f158fd51a11e3bcc966c756dd195a39"} Nov 01 23:01:31.929365 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:31.928975 5011 scope.go:122] "RemoveContainer" containerID="92768429b3f0b69587257e3da0d606a65e8b74fcfb37042a5aee1152e0dabe3e" Nov 01 23:01:32.275848 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.275751 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 23:01:32.431012 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.430975 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89a5b17f-1549-470c-8968-01b013df9417-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89a5b17f-1549-470c-8968-01b013df9417-istio-podinfo\") pod \"89a5b17f-1549-470c-8968-01b013df9417\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " Nov 01 23:01:32.431217 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.431022 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-credential-socket\") pod \"89a5b17f-1549-470c-8968-01b013df9417\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " Nov 01 23:01:32.431217 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.431153 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-workload-certs\") pod \"89a5b17f-1549-470c-8968-01b013df9417\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " Nov 01 23:01:32.431217 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.431190 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89a5b17f-1549-470c-8968-01b013df9417-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89a5b17f-1549-470c-8968-01b013df9417-istiod-ca-cert\") pod \"89a5b17f-1549-470c-8968-01b013df9417\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " Nov 01 23:01:32.431387 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.431251 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-istio-data\") pod \"89a5b17f-1549-470c-8968-01b013df9417\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " Nov 01 23:01:32.431444 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.431346 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-credential-socket" pod "89a5b17f-1549-470c-8968-01b013df9417" (UID: "89a5b17f-1549-470c-8968-01b013df9417"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:32.431444 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.431381 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-workload-socket\") pod \"89a5b17f-1549-470c-8968-01b013df9417\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " Nov 01 23:01:32.431556 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.431428 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-workload-certs" pod "89a5b17f-1549-470c-8968-01b013df9417" (UID: "89a5b17f-1549-470c-8968-01b013df9417"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:32.431556 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.431516 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89a5b17f-1549-470c-8968-01b013df9417-kube-api-access-r9nmg\" (UniqueName: \"kubernetes.io/projected/89a5b17f-1549-470c-8968-01b013df9417-kube-api-access-r9nmg\") pod \"89a5b17f-1549-470c-8968-01b013df9417\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " Nov 01 23:01:32.431658 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.431557 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-istio-envoy\") pod \"89a5b17f-1549-470c-8968-01b013df9417\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " Nov 01 23:01:32.431658 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.431568 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-istio-data" pod "89a5b17f-1549-470c-8968-01b013df9417" (UID: "89a5b17f-1549-470c-8968-01b013df9417"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:32.431658 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.431603 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-workload-socket" pod "89a5b17f-1549-470c-8968-01b013df9417" (UID: "89a5b17f-1549-470c-8968-01b013df9417"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:32.431658 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.431608 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89a5b17f-1549-470c-8968-01b013df9417-istiod-ca-cert" pod "89a5b17f-1549-470c-8968-01b013df9417" (UID: "89a5b17f-1549-470c-8968-01b013df9417"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:01:32.431658 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.431647 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89a5b17f-1549-470c-8968-01b013df9417-istio-token\" (UniqueName: \"kubernetes.io/projected/89a5b17f-1549-470c-8968-01b013df9417-istio-token\") pod \"89a5b17f-1549-470c-8968-01b013df9417\" (UID: \"89a5b17f-1549-470c-8968-01b013df9417\") " Nov 01 23:01:32.432351 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.432279 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89a5b17f-1549-470c-8968-01b013df9417-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:32.432351 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.432300 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:32.432351 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.432310 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:32.432351 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.432318 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:32.432351 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.432326 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:32.433434 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.433407 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89a5b17f-1549-470c-8968-01b013df9417-istio-podinfo" pod "89a5b17f-1549-470c-8968-01b013df9417" (UID: "89a5b17f-1549-470c-8968-01b013df9417"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:01:32.433906 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.433883 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89a5b17f-1549-470c-8968-01b013df9417-istio-token" pod "89a5b17f-1549-470c-8968-01b013df9417" (UID: "89a5b17f-1549-470c-8968-01b013df9417"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:32.434385 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.434364 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-istio-envoy" pod "89a5b17f-1549-470c-8968-01b013df9417" (UID: "89a5b17f-1549-470c-8968-01b013df9417"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:32.434385 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.434375 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89a5b17f-1549-470c-8968-01b013df9417-kube-api-access-r9nmg" pod "89a5b17f-1549-470c-8968-01b013df9417" (UID: "89a5b17f-1549-470c-8968-01b013df9417"). InnerVolumeSpecName "kube-api-access-r9nmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:32.533609 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.533519 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89a5b17f-1549-470c-8968-01b013df9417-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:32.533609 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.533552 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89a5b17f-1549-470c-8968-01b013df9417-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:32.533609 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.533564 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9nmg\" (UniqueName: \"kubernetes.io/projected/89a5b17f-1549-470c-8968-01b013df9417-kube-api-access-r9nmg\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:32.533609 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.533575 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89a5b17f-1549-470c-8968-01b013df9417-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:32.938013 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.937977 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" event={"ID":"89a5b17f-1549-470c-8968-01b013df9417","Type":"ContainerDied","Data":"ffd30464ba6874316fb19891a12058938ab3a6002c82f75c0eef63f6a0a04114"} Nov 01 23:01:32.938448 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.938049 5011 scope.go:122] "RemoveContainer" containerID="a1466adfa6bed6f0484768245c91d5f57f158fd51a11e3bcc966c756dd195a39" Nov 01 23:01:32.938448 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.937987 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs" Nov 01 23:01:32.965603 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.965572 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs"] Nov 01 23:01:32.970966 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:32.970937 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-xkvgs"] Nov 01 23:01:33.748675 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:33.748642 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89a5b17f-1549-470c-8968-01b013df9417" path="/var/lib/kubelet/pods/89a5b17f-1549-470c-8968-01b013df9417/volumes" Nov 01 23:01:41.554198 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:41.554159 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc"] Nov 01 23:01:41.554710 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:41.554392 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" podUID="7e14ab9c-363b-4f66-a585-3875dfaa45b6" containerName="istio-proxy" containerID="cri-o://20cef8e5a3ffb47b4b73b9dfa5f58408b1d8d7588eac30b4d72e67b0fb65de28" gracePeriod=30 Nov 01 23:01:45.737042 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:45.737006 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:01:45.737541 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:45.737078 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" podUID="7e14ab9c-363b-4f66-a585-3875dfaa45b6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:01:46.880145 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:46.880114 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 23:01:46.973030 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:01:46.972964 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:01:47.054993 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.054956 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e14ab9c-363b-4f66-a585-3875dfaa45b6-kube-api-access-z8tnl\" (UniqueName: \"kubernetes.io/projected/7e14ab9c-363b-4f66-a585-3875dfaa45b6-kube-api-access-z8tnl\") pod \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " Nov 01 23:01:47.055226 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.055051 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istio-envoy\") pod \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " Nov 01 23:01:47.055226 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.055137 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istiod-ca-cert\") pod \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " Nov 01 23:01:47.055226 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.055182 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istio-token\" (UniqueName: \"kubernetes.io/projected/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istio-token\") pod \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " Nov 01 23:01:47.055402 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.055327 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istio-podinfo\") pod \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " Nov 01 23:01:47.055465 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.055433 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-workload-socket\") pod \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " Nov 01 23:01:47.055529 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.055476 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istio-data\") pod \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " Nov 01 23:01:47.055585 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.055569 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-workload-certs\") pod \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " Nov 01 23:01:47.055650 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.055605 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-credential-socket\") pod \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\" (UID: \"7e14ab9c-363b-4f66-a585-3875dfaa45b6\") " Nov 01 23:01:47.055739 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.055703 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istiod-ca-cert" pod "7e14ab9c-363b-4f66-a585-3875dfaa45b6" (UID: "7e14ab9c-363b-4f66-a585-3875dfaa45b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:01:47.056061 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.056020 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istio-data" pod "7e14ab9c-363b-4f66-a585-3875dfaa45b6" (UID: "7e14ab9c-363b-4f66-a585-3875dfaa45b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:47.056314 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.056194 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-workload-socket" pod "7e14ab9c-363b-4f66-a585-3875dfaa45b6" (UID: "7e14ab9c-363b-4f66-a585-3875dfaa45b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:47.056314 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.056209 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-credential-socket" pod "7e14ab9c-363b-4f66-a585-3875dfaa45b6" (UID: "7e14ab9c-363b-4f66-a585-3875dfaa45b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:47.056724 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.056574 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.056724 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.056604 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.056724 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.056618 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.056724 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.056631 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.056724 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.056695 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-workload-certs" pod "7e14ab9c-363b-4f66-a585-3875dfaa45b6" (UID: "7e14ab9c-363b-4f66-a585-3875dfaa45b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:47.058465 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.058433 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istio-token" pod "7e14ab9c-363b-4f66-a585-3875dfaa45b6" (UID: "7e14ab9c-363b-4f66-a585-3875dfaa45b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:47.058579 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.058493 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istio-podinfo" pod "7e14ab9c-363b-4f66-a585-3875dfaa45b6" (UID: "7e14ab9c-363b-4f66-a585-3875dfaa45b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:01:47.061020 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.060992 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istio-envoy" pod "7e14ab9c-363b-4f66-a585-3875dfaa45b6" (UID: "7e14ab9c-363b-4f66-a585-3875dfaa45b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:47.061196 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.061041 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e14ab9c-363b-4f66-a585-3875dfaa45b6-kube-api-access-z8tnl" pod "7e14ab9c-363b-4f66-a585-3875dfaa45b6" (UID: "7e14ab9c-363b-4f66-a585-3875dfaa45b6"). InnerVolumeSpecName "kube-api-access-z8tnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:47.067959 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.067932 5011 generic.go:356] "Generic (PLEG): container finished" podID="7e14ab9c-363b-4f66-a585-3875dfaa45b6" containerID="20cef8e5a3ffb47b4b73b9dfa5f58408b1d8d7588eac30b4d72e67b0fb65de28" exitCode=0 Nov 01 23:01:47.068103 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.067994 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" Nov 01 23:01:47.068178 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.068163 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" event={"ID":"7e14ab9c-363b-4f66-a585-3875dfaa45b6","Type":"ContainerDied","Data":"20cef8e5a3ffb47b4b73b9dfa5f58408b1d8d7588eac30b4d72e67b0fb65de28"} Nov 01 23:01:47.068239 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.068205 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc" event={"ID":"7e14ab9c-363b-4f66-a585-3875dfaa45b6","Type":"ContainerDied","Data":"efed7206ef7242fc5fce1be79e73ab1b59a8c4642b31d59ba69eceadd8ddd7ee"} Nov 01 23:01:47.068239 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.068231 5011 scope.go:122] "RemoveContainer" containerID="20cef8e5a3ffb47b4b73b9dfa5f58408b1d8d7588eac30b4d72e67b0fb65de28" Nov 01 23:01:47.100590 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.100569 5011 scope.go:122] "RemoveContainer" containerID="20cef8e5a3ffb47b4b73b9dfa5f58408b1d8d7588eac30b4d72e67b0fb65de28" Nov 01 23:01:47.101008 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:01:47.100982 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20cef8e5a3ffb47b4b73b9dfa5f58408b1d8d7588eac30b4d72e67b0fb65de28\": container with ID starting with 20cef8e5a3ffb47b4b73b9dfa5f58408b1d8d7588eac30b4d72e67b0fb65de28 not found: ID does not exist" containerID="20cef8e5a3ffb47b4b73b9dfa5f58408b1d8d7588eac30b4d72e67b0fb65de28" Nov 01 23:01:47.101137 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.101016 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20cef8e5a3ffb47b4b73b9dfa5f58408b1d8d7588eac30b4d72e67b0fb65de28"} err="failed to get container status \"20cef8e5a3ffb47b4b73b9dfa5f58408b1d8d7588eac30b4d72e67b0fb65de28\": rpc error: code = NotFound desc = could not find container \"20cef8e5a3ffb47b4b73b9dfa5f58408b1d8d7588eac30b4d72e67b0fb65de28\": container with ID starting with 20cef8e5a3ffb47b4b73b9dfa5f58408b1d8d7588eac30b4d72e67b0fb65de28 not found: ID does not exist" Nov 01 23:01:47.114417 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.114390 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc"] Nov 01 23:01:47.132135 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.132052 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-mghtc"] Nov 01 23:01:47.167998 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.159465 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8tnl\" (UniqueName: \"kubernetes.io/projected/7e14ab9c-363b-4f66-a585-3875dfaa45b6-kube-api-access-z8tnl\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.167998 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.159510 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.167998 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.159528 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.167998 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.159542 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e14ab9c-363b-4f66-a585-3875dfaa45b6-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.167998 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.159556 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e14ab9c-363b-4f66-a585-3875dfaa45b6-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.759632 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:01:47.759596 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e14ab9c-363b-4f66-a585-3875dfaa45b6" path="/var/lib/kubelet/pods/7e14ab9c-363b-4f66-a585-3875dfaa45b6/volumes" Nov 01 23:02:13.705727 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:13.705692 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b"] Nov 01 23:02:13.706381 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:13.705979 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" podUID="00f9ac03-3cbe-4d33-9a63-feb3099cb6d3" containerName="discovery" containerID="cri-o://667350b76a7205dd92fdedba72efd0cc783167d285efad3d3707d4d68195b324" gracePeriod=30 Nov 01 23:02:14.085052 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:14.085026 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" Nov 01 23:02:14.194297 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:14.194264 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-istio-csr-ca-configmap\") pod \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\" (UID: \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\") " Nov 01 23:02:14.194477 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:14.194389 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-istio-csr-dns-cert\") pod \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\" (UID: \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\") " Nov 01 23:02:14.194477 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:14.194464 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-local-certs\") pod \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\" (UID: \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\") " Nov 01 23:02:14.194555 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:14.194498 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-istio-kubeconfig\") pod \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\" (UID: \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\") " Nov 01 23:02:14.194626 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:14.194609 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-kube-api-access-6wpk2\" (UniqueName: \"kubernetes.io/projected/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-kube-api-access-6wpk2\") pod \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\" (UID: \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\") " Nov 01 23:02:14.194685 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:14.194645 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-cacerts\" (UniqueName: \"kubernetes.io/secret/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-cacerts\") pod \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\" (UID: \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\") " Nov 01 23:02:14.194750 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:14.194688 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-istio-csr-ca-configmap" pod "00f9ac03-3cbe-4d33-9a63-feb3099cb6d3" (UID: "00f9ac03-3cbe-4d33-9a63-feb3099cb6d3"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:02:14.194808 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:14.194772 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-istio-token\" (UniqueName: \"kubernetes.io/projected/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-istio-token\") pod \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\" (UID: \"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3\") " Nov 01 23:02:14.196119 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:14.196088 5011 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-istio-csr-ca-configmap\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:14.197429 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:14.197397 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-kube-api-access-6wpk2" pod "00f9ac03-3cbe-4d33-9a63-feb3099cb6d3" (UID: "00f9ac03-3cbe-4d33-9a63-feb3099cb6d3"). InnerVolumeSpecName "kube-api-access-6wpk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:02:14.197656 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:14.197486 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-local-certs" pod "00f9ac03-3cbe-4d33-9a63-feb3099cb6d3" (UID: "00f9ac03-3cbe-4d33-9a63-feb3099cb6d3"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:14.197656 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:14.197508 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-cacerts" pod "00f9ac03-3cbe-4d33-9a63-feb3099cb6d3" (UID: "00f9ac03-3cbe-4d33-9a63-feb3099cb6d3"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:02:14.197656 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:14.197516 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-istio-csr-dns-cert" pod "00f9ac03-3cbe-4d33-9a63-feb3099cb6d3" (UID: "00f9ac03-3cbe-4d33-9a63-feb3099cb6d3"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:02:14.197656 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:14.197607 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-istio-token" pod "00f9ac03-3cbe-4d33-9a63-feb3099cb6d3" (UID: "00f9ac03-3cbe-4d33-9a63-feb3099cb6d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:02:14.197925 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:14.197707 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-istio-kubeconfig" pod "00f9ac03-3cbe-4d33-9a63-feb3099cb6d3" (UID: "00f9ac03-3cbe-4d33-9a63-feb3099cb6d3"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:02:14.297675 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:14.297630 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wpk2\" (UniqueName: \"kubernetes.io/projected/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-kube-api-access-6wpk2\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:14.297675 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:14.297666 5011 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-cacerts\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:14.297675 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:14.297681 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:14.298014 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:14.297695 5011 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-istio-csr-dns-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:14.298014 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:14.297708 5011 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-local-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:14.298014 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:14.297722 5011 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3-istio-kubeconfig\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:14.402435 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:14.402343 5011 generic.go:356] "Generic (PLEG): container finished" podID="00f9ac03-3cbe-4d33-9a63-feb3099cb6d3" containerID="667350b76a7205dd92fdedba72efd0cc783167d285efad3d3707d4d68195b324" exitCode=0 Nov 01 23:02:14.402435 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:14.402395 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" Nov 01 23:02:14.402435 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:14.402420 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" event={"ID":"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3","Type":"ContainerDied","Data":"667350b76a7205dd92fdedba72efd0cc783167d285efad3d3707d4d68195b324"} Nov 01 23:02:14.402670 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:14.402456 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b" event={"ID":"00f9ac03-3cbe-4d33-9a63-feb3099cb6d3","Type":"ContainerDied","Data":"a19cecfb1a0fbf8e2afb73ffc409f9bfa3c16b056112a93fbf414db4eea03ac1"} Nov 01 23:02:14.402670 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:14.402474 5011 scope.go:122] "RemoveContainer" containerID="667350b76a7205dd92fdedba72efd0cc783167d285efad3d3707d4d68195b324" Nov 01 23:02:14.415949 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:14.415922 5011 scope.go:122] "RemoveContainer" containerID="667350b76a7205dd92fdedba72efd0cc783167d285efad3d3707d4d68195b324" Nov 01 23:02:14.416222 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:02:14.416195 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"667350b76a7205dd92fdedba72efd0cc783167d285efad3d3707d4d68195b324\": container with ID starting with 667350b76a7205dd92fdedba72efd0cc783167d285efad3d3707d4d68195b324 not found: ID does not exist" containerID="667350b76a7205dd92fdedba72efd0cc783167d285efad3d3707d4d68195b324" Nov 01 23:02:14.416314 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:14.416227 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"667350b76a7205dd92fdedba72efd0cc783167d285efad3d3707d4d68195b324"} err="failed to get container status \"667350b76a7205dd92fdedba72efd0cc783167d285efad3d3707d4d68195b324\": rpc error: code = NotFound desc = could not find container \"667350b76a7205dd92fdedba72efd0cc783167d285efad3d3707d4d68195b324\": container with ID starting with 667350b76a7205dd92fdedba72efd0cc783167d285efad3d3707d4d68195b324 not found: ID does not exist" Nov 01 23:02:14.427464 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:14.427438 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b"] Nov 01 23:02:14.433898 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:14.433873 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hsr7b"] Nov 01 23:02:15.750098 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:15.750061 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00f9ac03-3cbe-4d33-9a63-feb3099cb6d3" path="/var/lib/kubelet/pods/00f9ac03-3cbe-4d33-9a63-feb3099cb6d3/volumes" Nov 01 23:02:17.025408 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:02:17.025359 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:02:19.525904 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:19.525854 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4"] Nov 01 23:02:19.526348 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:19.526071 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" podUID="5ffbc1a2-7268-4ab3-a81c-e04b42fc441f" containerName="istio-proxy" containerID="cri-o://eae3dcb08b31ef0b28e58073fac7c78478f9a0483077bdf49648f91bed39e39d" gracePeriod=30 Nov 01 23:02:24.910727 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:24.910693 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 23:02:25.105181 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.105144 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istio-podinfo\") pod \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " Nov 01 23:02:25.105386 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.105254 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-kube-api-access-ffvbn\" (UniqueName: \"kubernetes.io/projected/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-kube-api-access-ffvbn\") pod \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " Nov 01 23:02:25.105458 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.105408 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istio-envoy\") pod \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " Nov 01 23:02:25.105458 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.105445 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istio-token\" (UniqueName: \"kubernetes.io/projected/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istio-token\") pod \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " Nov 01 23:02:25.105570 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.105486 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-credential-socket\") pod \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " Nov 01 23:02:25.105625 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.105577 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istiod-ca-cert\") pod \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " Nov 01 23:02:25.105725 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.105708 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istio-data\") pod \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " Nov 01 23:02:25.105795 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.105742 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-workload-socket\") pod \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " Nov 01 23:02:25.105870 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.105805 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-workload-certs\") pod \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\" (UID: \"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f\") " Nov 01 23:02:25.106021 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.105993 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istio-data" pod "5ffbc1a2-7268-4ab3-a81c-e04b42fc441f" (UID: "5ffbc1a2-7268-4ab3-a81c-e04b42fc441f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:25.106092 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.106026 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istiod-ca-cert" pod "5ffbc1a2-7268-4ab3-a81c-e04b42fc441f" (UID: "5ffbc1a2-7268-4ab3-a81c-e04b42fc441f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:02:25.106205 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.106186 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-workload-socket" pod "5ffbc1a2-7268-4ab3-a81c-e04b42fc441f" (UID: "5ffbc1a2-7268-4ab3-a81c-e04b42fc441f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:25.106284 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.106264 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-workload-certs" pod "5ffbc1a2-7268-4ab3-a81c-e04b42fc441f" (UID: "5ffbc1a2-7268-4ab3-a81c-e04b42fc441f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:25.106479 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.106457 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-credential-socket" pod "5ffbc1a2-7268-4ab3-a81c-e04b42fc441f" (UID: "5ffbc1a2-7268-4ab3-a81c-e04b42fc441f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:25.106852 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.106799 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:25.106949 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.106821 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:25.106949 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.106875 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:25.106949 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.106890 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:25.108006 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.107976 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-kube-api-access-ffvbn" pod "5ffbc1a2-7268-4ab3-a81c-e04b42fc441f" (UID: "5ffbc1a2-7268-4ab3-a81c-e04b42fc441f"). InnerVolumeSpecName "kube-api-access-ffvbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:02:25.108257 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.108231 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istio-podinfo" pod "5ffbc1a2-7268-4ab3-a81c-e04b42fc441f" (UID: "5ffbc1a2-7268-4ab3-a81c-e04b42fc441f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:02:25.108341 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.108251 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istio-token" pod "5ffbc1a2-7268-4ab3-a81c-e04b42fc441f" (UID: "5ffbc1a2-7268-4ab3-a81c-e04b42fc441f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:02:25.108382 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.108357 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istio-envoy" pod "5ffbc1a2-7268-4ab3-a81c-e04b42fc441f" (UID: "5ffbc1a2-7268-4ab3-a81c-e04b42fc441f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:25.208496 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.208452 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:25.208496 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.208495 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffvbn\" (UniqueName: \"kubernetes.io/projected/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-kube-api-access-ffvbn\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:25.208768 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.208513 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:25.208768 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.208527 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:25.208768 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.208540 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:25.548005 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.547907 5011 generic.go:356] "Generic (PLEG): container finished" podID="5ffbc1a2-7268-4ab3-a81c-e04b42fc441f" containerID="eae3dcb08b31ef0b28e58073fac7c78478f9a0483077bdf49648f91bed39e39d" exitCode=0 Nov 01 23:02:25.548185 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.548152 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" event={"ID":"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f","Type":"ContainerDied","Data":"eae3dcb08b31ef0b28e58073fac7c78478f9a0483077bdf49648f91bed39e39d"} Nov 01 23:02:25.548305 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.548188 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" Nov 01 23:02:25.548305 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.548265 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4" event={"ID":"5ffbc1a2-7268-4ab3-a81c-e04b42fc441f","Type":"ContainerDied","Data":"360d224eb1111977543f33e967cdd7c31a181e9a15e2f993f14567a9edf3de12"} Nov 01 23:02:25.548305 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.548297 5011 scope.go:122] "RemoveContainer" containerID="eae3dcb08b31ef0b28e58073fac7c78478f9a0483077bdf49648f91bed39e39d" Nov 01 23:02:25.564264 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.564241 5011 scope.go:122] "RemoveContainer" containerID="eae3dcb08b31ef0b28e58073fac7c78478f9a0483077bdf49648f91bed39e39d" Nov 01 23:02:25.564589 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:02:25.564561 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eae3dcb08b31ef0b28e58073fac7c78478f9a0483077bdf49648f91bed39e39d\": container with ID starting with eae3dcb08b31ef0b28e58073fac7c78478f9a0483077bdf49648f91bed39e39d not found: ID does not exist" containerID="eae3dcb08b31ef0b28e58073fac7c78478f9a0483077bdf49648f91bed39e39d" Nov 01 23:02:25.564689 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.564596 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eae3dcb08b31ef0b28e58073fac7c78478f9a0483077bdf49648f91bed39e39d"} err="failed to get container status \"eae3dcb08b31ef0b28e58073fac7c78478f9a0483077bdf49648f91bed39e39d\": rpc error: code = NotFound desc = could not find container \"eae3dcb08b31ef0b28e58073fac7c78478f9a0483077bdf49648f91bed39e39d\": container with ID starting with eae3dcb08b31ef0b28e58073fac7c78478f9a0483077bdf49648f91bed39e39d not found: ID does not exist" Nov 01 23:02:25.579117 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.579085 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4"] Nov 01 23:02:25.582952 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.582930 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7jkf4"] Nov 01 23:02:25.749224 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:25.749186 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ffbc1a2-7268-4ab3-a81c-e04b42fc441f" path="/var/lib/kubelet/pods/5ffbc1a2-7268-4ab3-a81c-e04b42fc441f/volumes" Nov 01 23:02:28.543796 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.543758 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj"] Nov 01 23:02:28.545105 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.545079 5011 cpu_manager.go:414] "containerMap: removing container" podUID="7e14ab9c-363b-4f66-a585-3875dfaa45b6" containerName="istio-proxy" Nov 01 23:02:28.545289 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.545263 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e14ab9c-363b-4f66-a585-3875dfaa45b6" containerName="istio-proxy" Nov 01 23:02:28.545483 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.545314 5011 cpu_manager.go:414] "containerMap: removing container" podUID="665520e7-d3f3-4dd9-a7a5-3992fd827544" containerName="istio-proxy" Nov 01 23:02:28.545483 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.545324 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="665520e7-d3f3-4dd9-a7a5-3992fd827544" containerName="istio-proxy" Nov 01 23:02:28.545483 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.545352 5011 cpu_manager.go:414] "containerMap: removing container" podUID="89a5b17f-1549-470c-8968-01b013df9417" containerName="istio-proxy" Nov 01 23:02:28.545483 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.545359 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89a5b17f-1549-470c-8968-01b013df9417" containerName="istio-proxy" Nov 01 23:02:28.545483 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.545391 5011 cpu_manager.go:414] "containerMap: removing container" podUID="5ffbc1a2-7268-4ab3-a81c-e04b42fc441f" containerName="istio-proxy" Nov 01 23:02:28.545483 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.545398 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ffbc1a2-7268-4ab3-a81c-e04b42fc441f" containerName="istio-proxy" Nov 01 23:02:28.545483 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.545415 5011 cpu_manager.go:414] "containerMap: removing container" podUID="89a5b17f-1549-470c-8968-01b013df9417" containerName="istio-proxy" Nov 01 23:02:28.545483 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.545422 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89a5b17f-1549-470c-8968-01b013df9417" containerName="istio-proxy" Nov 01 23:02:28.545483 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.545440 5011 cpu_manager.go:414] "containerMap: removing container" podUID="deb96044-ee0a-4050-8076-ebce2c1d990a" containerName="istio-proxy" Nov 01 23:02:28.545483 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.545447 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deb96044-ee0a-4050-8076-ebce2c1d990a" containerName="istio-proxy" Nov 01 23:02:28.545483 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.545470 5011 cpu_manager.go:414] "containerMap: removing container" podUID="deb96044-ee0a-4050-8076-ebce2c1d990a" containerName="istio-proxy" Nov 01 23:02:28.545483 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.545476 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deb96044-ee0a-4050-8076-ebce2c1d990a" containerName="istio-proxy" Nov 01 23:02:28.545483 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.545497 5011 cpu_manager.go:414] "containerMap: removing container" podUID="00f9ac03-3cbe-4d33-9a63-feb3099cb6d3" containerName="discovery" Nov 01 23:02:28.545483 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.545504 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00f9ac03-3cbe-4d33-9a63-feb3099cb6d3" containerName="discovery" Nov 01 23:02:28.546364 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.545678 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="deb96044-ee0a-4050-8076-ebce2c1d990a" containerName="istio-proxy" Nov 01 23:02:28.546364 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.545693 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="00f9ac03-3cbe-4d33-9a63-feb3099cb6d3" containerName="discovery" Nov 01 23:02:28.546364 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.545711 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e14ab9c-363b-4f66-a585-3875dfaa45b6" containerName="istio-proxy" Nov 01 23:02:28.546364 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.545723 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="665520e7-d3f3-4dd9-a7a5-3992fd827544" containerName="istio-proxy" Nov 01 23:02:28.546364 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.545733 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="89a5b17f-1549-470c-8968-01b013df9417" containerName="istio-proxy" Nov 01 23:02:28.546364 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.545749 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ffbc1a2-7268-4ab3-a81c-e04b42fc441f" containerName="istio-proxy" Nov 01 23:02:28.546364 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.545761 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="89a5b17f-1549-470c-8968-01b013df9417" containerName="istio-proxy" Nov 01 23:02:28.546364 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.545772 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="deb96044-ee0a-4050-8076-ebce2c1d990a" containerName="istio-proxy" Nov 01 23:02:28.546364 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.545782 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="665520e7-d3f3-4dd9-a7a5-3992fd827544" containerName="istio-proxy" Nov 01 23:02:28.551866 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.551805 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:28.562685 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.562646 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-337dee-openshift-default-dockercfg-tqjnv\"" Nov 01 23:02:28.566498 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.566465 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj"] Nov 01 23:02:28.592661 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.589094 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-workload-socket\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:28.593001 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.592975 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-istio-envoy\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:28.593160 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.593045 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be85990a-36c7-4d36-8867-28b4c83867d8-istio-token\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:28.593160 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.593075 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-istio-data\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:28.593160 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.593147 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x4dh\" (UniqueName: \"kubernetes.io/projected/be85990a-36c7-4d36-8867-28b4c83867d8-kube-api-access-9x4dh\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:28.593341 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.593175 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be85990a-36c7-4d36-8867-28b4c83867d8-istiod-ca-cert\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:28.593341 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.593213 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be85990a-36c7-4d36-8867-28b4c83867d8-istio-podinfo\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:28.593341 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.593249 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-credential-socket\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:28.593341 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.593280 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-workload-certs\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:28.695854 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.695791 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-istio-envoy\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:28.696056 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.695908 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be85990a-36c7-4d36-8867-28b4c83867d8-istio-token\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:28.696056 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.695958 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-istio-data\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:28.696056 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.696049 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x4dh\" (UniqueName: \"kubernetes.io/projected/be85990a-36c7-4d36-8867-28b4c83867d8-kube-api-access-9x4dh\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:28.696233 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.696083 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be85990a-36c7-4d36-8867-28b4c83867d8-istiod-ca-cert\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:28.696233 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.696135 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be85990a-36c7-4d36-8867-28b4c83867d8-istio-podinfo\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:28.696233 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.696187 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-credential-socket\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:28.696233 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.696228 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-workload-certs\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:28.696435 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.696366 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-workload-socket\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:28.711859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.696867 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-workload-socket\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:28.712786 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.712758 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be85990a-36c7-4d36-8867-28b4c83867d8-istiod-ca-cert\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:28.736658 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.713186 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-istio-data\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:28.736658 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.713923 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-credential-socket\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:28.736658 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.725187 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-workload-certs\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:28.779923 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.773332 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be85990a-36c7-4d36-8867-28b4c83867d8-istio-podinfo\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:28.779923 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.773332 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x4dh\" (UniqueName: \"kubernetes.io/projected/be85990a-36c7-4d36-8867-28b4c83867d8-kube-api-access-9x4dh\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:28.779923 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.773891 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be85990a-36c7-4d36-8867-28b4c83867d8-istio-token\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:28.789460 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.789429 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-istio-envoy\") pod \"e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:28.909683 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:28.909611 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:29.235726 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:29.235695 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj"] Nov 01 23:02:29.235892 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:02:29.235871 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe85990a_36c7_4d36_8867_28b4c83867d8.slice/crio-9ae83a96a2a665e2d73c5b1fd6203891d5047ac4dac29cd0a6a5de745dc0a4a7 WatchSource:0}: Error finding container 9ae83a96a2a665e2d73c5b1fd6203891d5047ac4dac29cd0a6a5de745dc0a4a7: Status 404 returned error can't find the container with id 9ae83a96a2a665e2d73c5b1fd6203891d5047ac4dac29cd0a6a5de745dc0a4a7 Nov 01 23:02:29.239706 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:29.239666 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:02:29.239867 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:29.239749 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:02:29.239867 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:29.239818 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:02:29.602786 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:29.602685 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" event={"ID":"be85990a-36c7-4d36-8867-28b4c83867d8","Type":"ContainerStarted","Data":"92340491393b1c93d8968984727b54f3cd6f2e591097756d6138434964ae3988"} Nov 01 23:02:29.602786 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:29.602760 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" event={"ID":"be85990a-36c7-4d36-8867-28b4c83867d8","Type":"ContainerStarted","Data":"9ae83a96a2a665e2d73c5b1fd6203891d5047ac4dac29cd0a6a5de745dc0a4a7"} Nov 01 23:02:29.624622 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:29.624568 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" podStartSLOduration=1.624552875 podStartE2EDuration="1.624552875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:02:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:02:29.621350416 +0000 UTC m=+2318.025941047" watchObservedRunningTime="2026-11-01 23:02:29.624552875 +0000 UTC m=+2318.029143506" Nov 01 23:02:29.910223 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:29.910138 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:30.916483 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:30.916454 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:31.651500 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:31.649117 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:31.651500 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:31.650802 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:02:33.186474 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:33.186439 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-8545866b6c-79ndj"] Nov 01 23:02:33.187963 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:33.187935 5011 cpu_manager.go:414] "containerMap: removing container" podUID="665520e7-d3f3-4dd9-a7a5-3992fd827544" containerName="istio-proxy" Nov 01 23:02:33.187963 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:33.187960 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="665520e7-d3f3-4dd9-a7a5-3992fd827544" containerName="istio-proxy" Nov 01 23:02:33.192220 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:33.192191 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-8545866b6c-79ndj" Nov 01 23:02:33.218472 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:33.206117 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-337dee-dockercfg-8b4k8\"" Nov 01 23:02:33.218472 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:33.206178 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-337dee\"" Nov 01 23:02:33.218472 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:33.212249 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-8545866b6c-79ndj"] Nov 01 23:02:33.326819 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:33.326749 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5150a4c7-ec15-4b26-b21a-9177c0e8f75c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-337dee-8545866b6c-79ndj\" (UID: \"5150a4c7-ec15-4b26-b21a-9177c0e8f75c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-8545866b6c-79ndj" Nov 01 23:02:33.330303 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:33.330138 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtqz4\" (UniqueName: \"kubernetes.io/projected/5150a4c7-ec15-4b26-b21a-9177c0e8f75c-kube-api-access-rtqz4\") pod \"payload-pre-processing-e2e-ipp-a-337dee-8545866b6c-79ndj\" (UID: \"5150a4c7-ec15-4b26-b21a-9177c0e8f75c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-8545866b6c-79ndj" Nov 01 23:02:33.448115 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:33.446456 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtqz4\" (UniqueName: \"kubernetes.io/projected/5150a4c7-ec15-4b26-b21a-9177c0e8f75c-kube-api-access-rtqz4\") pod \"payload-pre-processing-e2e-ipp-a-337dee-8545866b6c-79ndj\" (UID: \"5150a4c7-ec15-4b26-b21a-9177c0e8f75c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-8545866b6c-79ndj" Nov 01 23:02:33.448115 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:33.446771 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5150a4c7-ec15-4b26-b21a-9177c0e8f75c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-337dee-8545866b6c-79ndj\" (UID: \"5150a4c7-ec15-4b26-b21a-9177c0e8f75c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-8545866b6c-79ndj" Nov 01 23:02:33.449235 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:33.449205 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5150a4c7-ec15-4b26-b21a-9177c0e8f75c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-337dee-8545866b6c-79ndj\" (UID: \"5150a4c7-ec15-4b26-b21a-9177c0e8f75c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-8545866b6c-79ndj" Nov 01 23:02:33.466657 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:33.464324 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtqz4\" (UniqueName: \"kubernetes.io/projected/5150a4c7-ec15-4b26-b21a-9177c0e8f75c-kube-api-access-rtqz4\") pod \"payload-pre-processing-e2e-ipp-a-337dee-8545866b6c-79ndj\" (UID: \"5150a4c7-ec15-4b26-b21a-9177c0e8f75c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-8545866b6c-79ndj" Nov 01 23:02:33.691059 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:33.691029 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-8545866b6c-79ndj" Nov 01 23:02:34.024309 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:34.024276 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-8545866b6c-79ndj"] Nov 01 23:02:34.025436 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:02:34.025405 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5150a4c7_ec15_4b26_b21a_9177c0e8f75c.slice/crio-ad367c8aee2047ad1f83d993294c9dee3d106ebae6ab44ca2762fa3ce5d2a756 WatchSource:0}: Error finding container ad367c8aee2047ad1f83d993294c9dee3d106ebae6ab44ca2762fa3ce5d2a756: Status 404 returned error can't find the container with id ad367c8aee2047ad1f83d993294c9dee3d106ebae6ab44ca2762fa3ce5d2a756 Nov 01 23:02:34.672299 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:34.672260 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-8545866b6c-79ndj" event={"ID":"5150a4c7-ec15-4b26-b21a-9177c0e8f75c","Type":"ContainerStarted","Data":"481ef94a4ebd74258147858a6f2b57c28a015a096510e059e88da435370da981"} Nov 01 23:02:34.672756 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:34.672312 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-8545866b6c-79ndj" event={"ID":"5150a4c7-ec15-4b26-b21a-9177c0e8f75c","Type":"ContainerStarted","Data":"ad367c8aee2047ad1f83d993294c9dee3d106ebae6ab44ca2762fa3ce5d2a756"} Nov 01 23:02:34.672756 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:34.672480 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-8545866b6c-79ndj" Nov 01 23:02:34.696338 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:34.696285 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-8545866b6c-79ndj" podStartSLOduration=1.6962719800000001 podStartE2EDuration="1.69627198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:02:34.692795726 +0000 UTC m=+2323.097386357" watchObservedRunningTime="2026-11-01 23:02:34.69627198 +0000 UTC m=+2323.100862611" Nov 01 23:02:45.694300 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:45.694270 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-8545866b6c-79ndj" Nov 01 23:02:46.964009 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:02:46.963969 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:02:48.048872 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:02:48.048812 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:03:01.038035 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:01.037990 5011 generic.go:356] "Generic (PLEG): container finished" podID="55ecde9f-2ac4-4791-ada0-184f9d6141f8" containerID="61dcae3adeec8abf26fe5998cbd979dde6ea70f158b6ac5014f02b02dbac20a0" exitCode=0 Nov 01 23:03:01.038662 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:01.038073 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" event={"ID":"55ecde9f-2ac4-4791-ada0-184f9d6141f8","Type":"ContainerDied","Data":"61dcae3adeec8abf26fe5998cbd979dde6ea70f158b6ac5014f02b02dbac20a0"} Nov 01 23:03:01.038662 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:01.038646 5011 scope.go:122] "RemoveContainer" containerID="61dcae3adeec8abf26fe5998cbd979dde6ea70f158b6ac5014f02b02dbac20a0" Nov 01 23:03:01.041678 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:01.041636 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:03:01.041871 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:01.041737 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:03:01.041871 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:01.041792 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:03:02.069909 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:02.069874 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" event={"ID":"55ecde9f-2ac4-4791-ada0-184f9d6141f8","Type":"ContainerStarted","Data":"db051a13764c72d72b26b4f5a2333d3acc9c3ba66e31723d2d7becd52a136cfc"} Nov 01 23:03:02.569876 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:02.569840 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 23:03:02.571521 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:02.571497 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 01 23:03:02.571660 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:02.571544 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" podUID="55ecde9f-2ac4-4791-ada0-184f9d6141f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 01 23:03:03.569868 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:03.569814 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 01 23:03:03.570274 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:03.569895 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" podUID="55ecde9f-2ac4-4791-ada0-184f9d6141f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 01 23:03:04.107181 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:04.107146 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 23:03:04.570855 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:04.570241 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 01 23:03:04.570855 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:04.570307 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" podUID="55ecde9f-2ac4-4791-ada0-184f9d6141f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 01 23:03:05.570333 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:05.570294 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 01 23:03:05.570529 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:05.570350 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" podUID="55ecde9f-2ac4-4791-ada0-184f9d6141f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 01 23:03:06.570677 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:06.570632 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 01 23:03:06.571197 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:06.570715 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" podUID="55ecde9f-2ac4-4791-ada0-184f9d6141f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 01 23:03:07.569757 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:07.569716 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Nov 01 23:03:07.569998 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:07.569784 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" podUID="55ecde9f-2ac4-4791-ada0-184f9d6141f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Nov 01 23:03:08.575962 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:08.575928 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 23:03:08.577687 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:08.577662 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 23:03:12.289970 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:12.289931 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-8545866b6c-79ndj"] Nov 01 23:03:16.961031 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:03:16.960991 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:03:17.291164 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:17.291075 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-8545866b6c-79ndj" podUID="5150a4c7-ec15-4b26-b21a-9177c0e8f75c" containerName="payload-pre-processing" containerID="cri-o://481ef94a4ebd74258147858a6f2b57c28a015a096510e059e88da435370da981" gracePeriod=25 Nov 01 23:03:17.580460 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:17.580427 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-8545866b6c-79ndj" Nov 01 23:03:17.657663 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:17.657629 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5150a4c7-ec15-4b26-b21a-9177c0e8f75c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5150a4c7-ec15-4b26-b21a-9177c0e8f75c-plugins-config-volume\") pod \"5150a4c7-ec15-4b26-b21a-9177c0e8f75c\" (UID: \"5150a4c7-ec15-4b26-b21a-9177c0e8f75c\") " Nov 01 23:03:17.657848 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:17.657801 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5150a4c7-ec15-4b26-b21a-9177c0e8f75c-kube-api-access-rtqz4\" (UniqueName: \"kubernetes.io/projected/5150a4c7-ec15-4b26-b21a-9177c0e8f75c-kube-api-access-rtqz4\") pod \"5150a4c7-ec15-4b26-b21a-9177c0e8f75c\" (UID: \"5150a4c7-ec15-4b26-b21a-9177c0e8f75c\") " Nov 01 23:03:17.658098 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:17.658075 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5150a4c7-ec15-4b26-b21a-9177c0e8f75c-plugins-config-volume" pod "5150a4c7-ec15-4b26-b21a-9177c0e8f75c" (UID: "5150a4c7-ec15-4b26-b21a-9177c0e8f75c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:03:17.658450 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:17.658428 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5150a4c7-ec15-4b26-b21a-9177c0e8f75c-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:17.660059 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:17.660036 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5150a4c7-ec15-4b26-b21a-9177c0e8f75c-kube-api-access-rtqz4" pod "5150a4c7-ec15-4b26-b21a-9177c0e8f75c" (UID: "5150a4c7-ec15-4b26-b21a-9177c0e8f75c"). InnerVolumeSpecName "kube-api-access-rtqz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:03:17.760530 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:17.760498 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtqz4\" (UniqueName: \"kubernetes.io/projected/5150a4c7-ec15-4b26-b21a-9177c0e8f75c-kube-api-access-rtqz4\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:18.299409 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:18.299373 5011 generic.go:356] "Generic (PLEG): container finished" podID="5150a4c7-ec15-4b26-b21a-9177c0e8f75c" containerID="481ef94a4ebd74258147858a6f2b57c28a015a096510e059e88da435370da981" exitCode=0 Nov 01 23:03:18.299916 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:18.299420 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-8545866b6c-79ndj" event={"ID":"5150a4c7-ec15-4b26-b21a-9177c0e8f75c","Type":"ContainerDied","Data":"481ef94a4ebd74258147858a6f2b57c28a015a096510e059e88da435370da981"} Nov 01 23:03:18.299916 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:18.299424 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-8545866b6c-79ndj" Nov 01 23:03:18.299916 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:18.299461 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-8545866b6c-79ndj" event={"ID":"5150a4c7-ec15-4b26-b21a-9177c0e8f75c","Type":"ContainerDied","Data":"ad367c8aee2047ad1f83d993294c9dee3d106ebae6ab44ca2762fa3ce5d2a756"} Nov 01 23:03:18.299916 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:18.299482 5011 scope.go:122] "RemoveContainer" containerID="481ef94a4ebd74258147858a6f2b57c28a015a096510e059e88da435370da981" Nov 01 23:03:18.310311 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:18.310288 5011 scope.go:122] "RemoveContainer" containerID="481ef94a4ebd74258147858a6f2b57c28a015a096510e059e88da435370da981" Nov 01 23:03:18.310642 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:03:18.310621 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"481ef94a4ebd74258147858a6f2b57c28a015a096510e059e88da435370da981\": container with ID starting with 481ef94a4ebd74258147858a6f2b57c28a015a096510e059e88da435370da981 not found: ID does not exist" containerID="481ef94a4ebd74258147858a6f2b57c28a015a096510e059e88da435370da981" Nov 01 23:03:18.310692 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:18.310653 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"481ef94a4ebd74258147858a6f2b57c28a015a096510e059e88da435370da981"} err="failed to get container status \"481ef94a4ebd74258147858a6f2b57c28a015a096510e059e88da435370da981\": rpc error: code = NotFound desc = could not find container \"481ef94a4ebd74258147858a6f2b57c28a015a096510e059e88da435370da981\": container with ID starting with 481ef94a4ebd74258147858a6f2b57c28a015a096510e059e88da435370da981 not found: ID does not exist" Nov 01 23:03:18.318969 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:18.318940 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-8545866b6c-79ndj"] Nov 01 23:03:18.322970 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:18.322948 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-8545866b6c-79ndj"] Nov 01 23:03:19.748554 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:19.748521 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5150a4c7-ec15-4b26-b21a-9177c0e8f75c" path="/var/lib/kubelet/pods/5150a4c7-ec15-4b26-b21a-9177c0e8f75c/volumes" Nov 01 23:03:22.689369 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:22.689334 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-64b196-5c56f9cc87-sqg9q"] Nov 01 23:03:27.690665 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:27.690618 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-64b196-5c56f9cc87-sqg9q" podUID="e42f2aff-b236-41b5-842e-0e4cb93f8b92" containerName="payload-processing" containerID="cri-o://7cd9b9428bbe35c031ed7caff3c6e7dd6570c2b55ab5bab7999d7831b74b1165" gracePeriod=25 Nov 01 23:03:27.982309 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:27.982277 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-64b196-5c56f9cc87-sqg9q" Nov 01 23:03:28.011003 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:28.010972 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e42f2aff-b236-41b5-842e-0e4cb93f8b92-kube-api-access-wzt57\" (UniqueName: \"kubernetes.io/projected/e42f2aff-b236-41b5-842e-0e4cb93f8b92-kube-api-access-wzt57\") pod \"e42f2aff-b236-41b5-842e-0e4cb93f8b92\" (UID: \"e42f2aff-b236-41b5-842e-0e4cb93f8b92\") " Nov 01 23:03:28.011200 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:28.011124 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e42f2aff-b236-41b5-842e-0e4cb93f8b92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e42f2aff-b236-41b5-842e-0e4cb93f8b92-plugins-config-volume\") pod \"e42f2aff-b236-41b5-842e-0e4cb93f8b92\" (UID: \"e42f2aff-b236-41b5-842e-0e4cb93f8b92\") " Nov 01 23:03:28.012186 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:28.012132 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e42f2aff-b236-41b5-842e-0e4cb93f8b92-plugins-config-volume" pod "e42f2aff-b236-41b5-842e-0e4cb93f8b92" (UID: "e42f2aff-b236-41b5-842e-0e4cb93f8b92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:03:28.014683 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:28.014381 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e42f2aff-b236-41b5-842e-0e4cb93f8b92-kube-api-access-wzt57" pod "e42f2aff-b236-41b5-842e-0e4cb93f8b92" (UID: "e42f2aff-b236-41b5-842e-0e4cb93f8b92"). InnerVolumeSpecName "kube-api-access-wzt57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:03:28.113407 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:28.113307 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzt57\" (UniqueName: \"kubernetes.io/projected/e42f2aff-b236-41b5-842e-0e4cb93f8b92-kube-api-access-wzt57\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:28.113407 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:28.113345 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e42f2aff-b236-41b5-842e-0e4cb93f8b92-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:28.401254 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:28.401164 5011 generic.go:356] "Generic (PLEG): container finished" podID="e42f2aff-b236-41b5-842e-0e4cb93f8b92" containerID="7cd9b9428bbe35c031ed7caff3c6e7dd6570c2b55ab5bab7999d7831b74b1165" exitCode=0 Nov 01 23:03:28.401254 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:28.401216 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-64b196-5c56f9cc87-sqg9q" Nov 01 23:03:28.401254 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:28.401228 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-64b196-5c56f9cc87-sqg9q" event={"ID":"e42f2aff-b236-41b5-842e-0e4cb93f8b92","Type":"ContainerDied","Data":"7cd9b9428bbe35c031ed7caff3c6e7dd6570c2b55ab5bab7999d7831b74b1165"} Nov 01 23:03:28.401501 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:28.401278 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-64b196-5c56f9cc87-sqg9q" event={"ID":"e42f2aff-b236-41b5-842e-0e4cb93f8b92","Type":"ContainerDied","Data":"9d04c093f836aa8e12a6857a76ede898188c26bce765478a49327e2f6d1371af"} Nov 01 23:03:28.401501 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:28.401301 5011 scope.go:122] "RemoveContainer" containerID="7cd9b9428bbe35c031ed7caff3c6e7dd6570c2b55ab5bab7999d7831b74b1165" Nov 01 23:03:28.412450 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:28.412432 5011 scope.go:122] "RemoveContainer" containerID="7cd9b9428bbe35c031ed7caff3c6e7dd6570c2b55ab5bab7999d7831b74b1165" Nov 01 23:03:28.412772 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:03:28.412753 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cd9b9428bbe35c031ed7caff3c6e7dd6570c2b55ab5bab7999d7831b74b1165\": container with ID starting with 7cd9b9428bbe35c031ed7caff3c6e7dd6570c2b55ab5bab7999d7831b74b1165 not found: ID does not exist" containerID="7cd9b9428bbe35c031ed7caff3c6e7dd6570c2b55ab5bab7999d7831b74b1165" Nov 01 23:03:28.412825 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:28.412782 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cd9b9428bbe35c031ed7caff3c6e7dd6570c2b55ab5bab7999d7831b74b1165"} err="failed to get container status \"7cd9b9428bbe35c031ed7caff3c6e7dd6570c2b55ab5bab7999d7831b74b1165\": rpc error: code = NotFound desc = could not find container \"7cd9b9428bbe35c031ed7caff3c6e7dd6570c2b55ab5bab7999d7831b74b1165\": container with ID starting with 7cd9b9428bbe35c031ed7caff3c6e7dd6570c2b55ab5bab7999d7831b74b1165 not found: ID does not exist" Nov 01 23:03:28.432220 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:28.432189 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-64b196-5c56f9cc87-sqg9q"] Nov 01 23:03:28.435743 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:28.435723 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-64b196-5c56f9cc87-sqg9q"] Nov 01 23:03:29.750513 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:29.750480 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e42f2aff-b236-41b5-842e-0e4cb93f8b92" path="/var/lib/kubelet/pods/e42f2aff-b236-41b5-842e-0e4cb93f8b92/volumes" Nov 01 23:03:44.502663 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.502563 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq"] Nov 01 23:03:44.535891 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.534718 5011 cpu_manager.go:414] "containerMap: removing container" podUID="e42f2aff-b236-41b5-842e-0e4cb93f8b92" containerName="payload-processing" Nov 01 23:03:44.535891 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.534751 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e42f2aff-b236-41b5-842e-0e4cb93f8b92" containerName="payload-processing" Nov 01 23:03:44.535891 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.534768 5011 cpu_manager.go:414] "containerMap: removing container" podUID="5150a4c7-ec15-4b26-b21a-9177c0e8f75c" containerName="payload-pre-processing" Nov 01 23:03:44.535891 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.534776 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5150a4c7-ec15-4b26-b21a-9177c0e8f75c" containerName="payload-pre-processing" Nov 01 23:03:44.535891 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.535352 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="e42f2aff-b236-41b5-842e-0e4cb93f8b92" containerName="payload-processing" Nov 01 23:03:44.535891 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.535375 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="5150a4c7-ec15-4b26-b21a-9177c0e8f75c" containerName="payload-pre-processing" Nov 01 23:03:44.585621 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.584959 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq"] Nov 01 23:03:44.588533 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.585944 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:44.705131 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.620181 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-ec0da7-openshift-default-dockercfg-dcrt6\"" Nov 01 23:03:44.736738 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.726051 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-workload-certs\") pod \"e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:44.753641 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.753570 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpbf4\" (UniqueName: \"kubernetes.io/projected/07a622eb-374c-400c-bb23-daccb27e6a3a-kube-api-access-qpbf4\") pod \"e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:44.754035 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.754011 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-workload-socket\") pod \"e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:44.754221 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.754205 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a622eb-374c-400c-bb23-daccb27e6a3a-istio-token\") pod \"e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:44.754344 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.754329 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a622eb-374c-400c-bb23-daccb27e6a3a-istiod-ca-cert\") pod \"e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:44.754468 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.754454 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-istio-envoy\") pod \"e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:44.754632 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.754617 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a622eb-374c-400c-bb23-daccb27e6a3a-istio-podinfo\") pod \"e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:44.754882 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.754862 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-istio-data\") pod \"e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:44.755106 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.755089 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-credential-socket\") pod \"e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:44.862385 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.862345 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpbf4\" (UniqueName: \"kubernetes.io/projected/07a622eb-374c-400c-bb23-daccb27e6a3a-kube-api-access-qpbf4\") pod \"e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:44.862773 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.862751 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-workload-socket\") pod \"e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:44.867961 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.867929 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a622eb-374c-400c-bb23-daccb27e6a3a-istio-token\") pod \"e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:44.868164 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.868145 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a622eb-374c-400c-bb23-daccb27e6a3a-istiod-ca-cert\") pod \"e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:44.868316 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.868299 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-istio-envoy\") pod \"e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:44.873309 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.868470 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a622eb-374c-400c-bb23-daccb27e6a3a-istio-podinfo\") pod \"e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:44.904369 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.904337 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-istio-data\") pod \"e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:44.905023 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.905002 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-credential-socket\") pod \"e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:44.905256 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.905242 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-workload-certs\") pod \"e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:44.905453 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.904870 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-istio-data\") pod \"e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:44.906041 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.905737 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-credential-socket\") pod \"e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:44.906176 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.877460 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a622eb-374c-400c-bb23-daccb27e6a3a-istiod-ca-cert\") pod \"e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:44.906288 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.867806 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-workload-socket\") pod \"e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:44.906386 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.905961 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-workload-certs\") pod \"e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:44.934457 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.934422 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-istio-envoy\") pod \"e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:44.936784 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.936757 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpbf4\" (UniqueName: \"kubernetes.io/projected/07a622eb-374c-400c-bb23-daccb27e6a3a-kube-api-access-qpbf4\") pod \"e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:44.953852 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.953806 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a622eb-374c-400c-bb23-daccb27e6a3a-istio-token\") pod \"e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:44.989900 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:44.982516 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a622eb-374c-400c-bb23-daccb27e6a3a-istio-podinfo\") pod \"e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:45.064901 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:45.060113 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:45.623066 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:45.623005 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq"] Nov 01 23:03:45.625224 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:03:45.625190 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a622eb_374c_400c_bb23_daccb27e6a3a.slice/crio-475604e9a2d21e175ac6702c1e9b75a02f1a0516af7367ad2760ad8df9d768c1 WatchSource:0}: Error finding container 475604e9a2d21e175ac6702c1e9b75a02f1a0516af7367ad2760ad8df9d768c1: Status 404 returned error can't find the container with id 475604e9a2d21e175ac6702c1e9b75a02f1a0516af7367ad2760ad8df9d768c1 Nov 01 23:03:45.629168 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:45.629125 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:03:45.629305 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:45.629212 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:03:45.629305 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:45.629242 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:03:45.800849 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:45.799319 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" event={"ID":"07a622eb-374c-400c-bb23-daccb27e6a3a","Type":"ContainerStarted","Data":"04f817f300e1124a616af657f8070286bb1ed67c1d73072fd5473dd7c279694b"} Nov 01 23:03:45.800849 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:45.799381 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" event={"ID":"07a622eb-374c-400c-bb23-daccb27e6a3a","Type":"ContainerStarted","Data":"475604e9a2d21e175ac6702c1e9b75a02f1a0516af7367ad2760ad8df9d768c1"} Nov 01 23:03:45.822295 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:45.822240 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" podStartSLOduration=1.8222228679999999 podStartE2EDuration="1.822222868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:03:45.821080218 +0000 UTC m=+2394.225670852" watchObservedRunningTime="2026-11-01 23:03:45.822222868 +0000 UTC m=+2394.226813502" Nov 01 23:03:46.064456 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:46.063999 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:46.064456 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:46.064126 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 01 23:03:46.064456 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:46.064187 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" podUID="07a622eb-374c-400c-bb23-daccb27e6a3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 01 23:03:46.988051 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:03:46.988016 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:03:47.071691 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:47.071662 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:47.814792 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:47.814757 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:47.815882 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:47.815863 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:03:49.567900 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:49.567856 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-6dc87574c-bhqsg"] Nov 01 23:03:49.576679 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:49.576649 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-6dc87574c-bhqsg" Nov 01 23:03:49.578906 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:49.578814 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-6dc87574c-bhqsg"] Nov 01 23:03:49.580960 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:49.580937 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ec0da7-dockercfg-c8b5v\"" Nov 01 23:03:49.581096 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:49.580954 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ec0da7\"" Nov 01 23:03:49.629413 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:49.629382 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9510c1fd-1773-4783-b86a-224f7c96c35c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ec0da7-6dc87574c-bhqsg\" (UID: \"9510c1fd-1773-4783-b86a-224f7c96c35c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-6dc87574c-bhqsg" Nov 01 23:03:49.630092 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:49.630066 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfhnx\" (UniqueName: \"kubernetes.io/projected/9510c1fd-1773-4783-b86a-224f7c96c35c-kube-api-access-wfhnx\") pod \"payload-pre-processing-e2e-ipp-b-ec0da7-6dc87574c-bhqsg\" (UID: \"9510c1fd-1773-4783-b86a-224f7c96c35c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-6dc87574c-bhqsg" Nov 01 23:03:49.831908 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:49.753646 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9510c1fd-1773-4783-b86a-224f7c96c35c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ec0da7-6dc87574c-bhqsg\" (UID: \"9510c1fd-1773-4783-b86a-224f7c96c35c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-6dc87574c-bhqsg" Nov 01 23:03:49.931042 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:49.799779 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9510c1fd-1773-4783-b86a-224f7c96c35c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ec0da7-6dc87574c-bhqsg\" (UID: \"9510c1fd-1773-4783-b86a-224f7c96c35c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-6dc87574c-bhqsg" Nov 01 23:03:49.931454 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:49.931420 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfhnx\" (UniqueName: \"kubernetes.io/projected/9510c1fd-1773-4783-b86a-224f7c96c35c-kube-api-access-wfhnx\") pod \"payload-pre-processing-e2e-ipp-b-ec0da7-6dc87574c-bhqsg\" (UID: \"9510c1fd-1773-4783-b86a-224f7c96c35c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-6dc87574c-bhqsg" Nov 01 23:03:49.982004 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:49.981971 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfhnx\" (UniqueName: \"kubernetes.io/projected/9510c1fd-1773-4783-b86a-224f7c96c35c-kube-api-access-wfhnx\") pod \"payload-pre-processing-e2e-ipp-b-ec0da7-6dc87574c-bhqsg\" (UID: \"9510c1fd-1773-4783-b86a-224f7c96c35c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-6dc87574c-bhqsg" Nov 01 23:03:50.196868 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:50.196815 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-6dc87574c-bhqsg" Nov 01 23:03:50.633035 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:50.632992 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-6dc87574c-bhqsg"] Nov 01 23:03:51.037690 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:51.037648 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-6dc87574c-bhqsg" event={"ID":"9510c1fd-1773-4783-b86a-224f7c96c35c","Type":"ContainerStarted","Data":"41e1ec004046c51ecc679f63279ee6bffb5593e23ee95d86ddb5f49144079b41"} Nov 01 23:03:51.038012 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:51.037988 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-6dc87574c-bhqsg" event={"ID":"9510c1fd-1773-4783-b86a-224f7c96c35c","Type":"ContainerStarted","Data":"9a72f55b800af012550051d653bbef8250bf91f569609275c50d20ff347dd20c"} Nov 01 23:03:51.039445 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:51.039420 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-6dc87574c-bhqsg" Nov 01 23:03:51.077020 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:51.076305 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-6dc87574c-bhqsg" podStartSLOduration=2.076284578 podStartE2EDuration="2.076284578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:03:51.068568499 +0000 UTC m=+2399.473159133" watchObservedRunningTime="2026-11-01 23:03:51.076284578 +0000 UTC m=+2399.480875215" Nov 01 23:03:56.267152 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:03:56.267115 5011 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@11-32195-0.service\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe85990a_36c7_4d36_8867_28b4c83867d8.slice/crio-92340491393b1c93d8968984727b54f3cd6f2e591097756d6138434964ae3988.scope\": RecentStats: unable to find data in memory cache]" Nov 01 23:03:57.092465 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:57.092432 5011 generic.go:356] "Generic (PLEG): container finished" podID="be85990a-36c7-4d36-8867-28b4c83867d8" containerID="92340491393b1c93d8968984727b54f3cd6f2e591097756d6138434964ae3988" exitCode=0 Nov 01 23:03:57.092651 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:57.092503 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" event={"ID":"be85990a-36c7-4d36-8867-28b4c83867d8","Type":"ContainerDied","Data":"92340491393b1c93d8968984727b54f3cd6f2e591097756d6138434964ae3988"} Nov 01 23:03:57.092971 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:57.092955 5011 scope.go:122] "RemoveContainer" containerID="92340491393b1c93d8968984727b54f3cd6f2e591097756d6138434964ae3988" Nov 01 23:03:57.094901 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:57.094869 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:03:57.094983 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:57.094949 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:03:57.095021 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:57.094982 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:03:58.102715 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:58.102683 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" event={"ID":"be85990a-36c7-4d36-8867-28b4c83867d8","Type":"ContainerStarted","Data":"bf9f10c16f9cdab3b85815869b4269f7d16cb3d55551c2a209f5befc81cb8d05"} Nov 01 23:03:58.911166 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:58.911130 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:03:58.911630 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:58.911608 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 01 23:03:58.911844 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:58.911801 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" podUID="be85990a-36c7-4d36-8867-28b4c83867d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 01 23:03:59.910934 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:59.910903 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 01 23:03:59.911377 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:03:59.910957 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" podUID="be85990a-36c7-4d36-8867-28b4c83867d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 01 23:04:00.910488 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:00.910449 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 01 23:04:00.910688 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:00.910508 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" podUID="be85990a-36c7-4d36-8867-28b4c83867d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 01 23:04:01.651984 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:01.651942 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:04:01.910324 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:01.910229 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 01 23:04:01.910324 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:01.910284 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" podUID="be85990a-36c7-4d36-8867-28b4c83867d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 01 23:04:02.910796 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:02.910760 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 01 23:04:02.911207 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:02.910846 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" podUID="be85990a-36c7-4d36-8867-28b4c83867d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 01 23:04:03.066693 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:03.066662 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-6dc87574c-bhqsg" Nov 01 23:04:04.296244 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:04.296190 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:04:04.298697 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:04.298677 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:04:13.318265 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:13.318228 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:04:16.964307 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:04:16.964258 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:04:24.439283 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:24.439225 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-648c48588c-bf8mt"] Nov 01 23:04:24.445736 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:24.445706 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-648c48588c-bf8mt" Nov 01 23:04:24.458048 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:24.457994 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8e3381\"" Nov 01 23:04:24.460745 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:24.460712 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-648c48588c-bf8mt"] Nov 01 23:04:24.497940 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:24.497900 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e3381-5df69964cc-c9r7x"] Nov 01 23:04:24.504206 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:24.504181 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-5df69964cc-c9r7x" Nov 01 23:04:24.512283 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:24.512249 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e3381-5df69964cc-c9r7x"] Nov 01 23:04:24.586856 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:24.571138 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8e3381-dockercfg-rqtnr\"" Nov 01 23:04:24.736976 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:24.708802 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgbhg\" (UniqueName: \"kubernetes.io/projected/2ec00952-a3a1-436f-935f-d26957dda8fe-kube-api-access-rgbhg\") pod \"payload-pre-processing-e2e-ipp-a-8e3381-648c48588c-bf8mt\" (UID: \"2ec00952-a3a1-436f-935f-d26957dda8fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-648c48588c-bf8mt" Nov 01 23:04:24.782341 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:24.782306 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmz52\" (UniqueName: \"kubernetes.io/projected/951af526-1d75-4b53-8e7f-d11210ffadd1-kube-api-access-lmz52\") pod \"payload-processing-e2e-ipp-a-8e3381-5df69964cc-c9r7x\" (UID: \"951af526-1d75-4b53-8e7f-d11210ffadd1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-5df69964cc-c9r7x" Nov 01 23:04:24.782638 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:24.782611 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec00952-a3a1-436f-935f-d26957dda8fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8e3381-648c48588c-bf8mt\" (UID: \"2ec00952-a3a1-436f-935f-d26957dda8fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-648c48588c-bf8mt" Nov 01 23:04:24.783079 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:24.783047 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/951af526-1d75-4b53-8e7f-d11210ffadd1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8e3381-5df69964cc-c9r7x\" (UID: \"951af526-1d75-4b53-8e7f-d11210ffadd1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-5df69964cc-c9r7x" Nov 01 23:04:24.884286 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:24.884251 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/951af526-1d75-4b53-8e7f-d11210ffadd1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8e3381-5df69964cc-c9r7x\" (UID: \"951af526-1d75-4b53-8e7f-d11210ffadd1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-5df69964cc-c9r7x" Nov 01 23:04:24.884471 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:24.884403 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgbhg\" (UniqueName: \"kubernetes.io/projected/2ec00952-a3a1-436f-935f-d26957dda8fe-kube-api-access-rgbhg\") pod \"payload-pre-processing-e2e-ipp-a-8e3381-648c48588c-bf8mt\" (UID: \"2ec00952-a3a1-436f-935f-d26957dda8fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-648c48588c-bf8mt" Nov 01 23:04:24.884471 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:24.884449 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmz52\" (UniqueName: \"kubernetes.io/projected/951af526-1d75-4b53-8e7f-d11210ffadd1-kube-api-access-lmz52\") pod \"payload-processing-e2e-ipp-a-8e3381-5df69964cc-c9r7x\" (UID: \"951af526-1d75-4b53-8e7f-d11210ffadd1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-5df69964cc-c9r7x" Nov 01 23:04:24.884620 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:24.884501 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec00952-a3a1-436f-935f-d26957dda8fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8e3381-648c48588c-bf8mt\" (UID: \"2ec00952-a3a1-436f-935f-d26957dda8fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-648c48588c-bf8mt" Nov 01 23:04:24.885149 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:24.885121 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/951af526-1d75-4b53-8e7f-d11210ffadd1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8e3381-5df69964cc-c9r7x\" (UID: \"951af526-1d75-4b53-8e7f-d11210ffadd1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-5df69964cc-c9r7x" Nov 01 23:04:24.885525 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:24.885494 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec00952-a3a1-436f-935f-d26957dda8fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8e3381-648c48588c-bf8mt\" (UID: \"2ec00952-a3a1-436f-935f-d26957dda8fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-648c48588c-bf8mt" Nov 01 23:04:24.896358 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:24.896332 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmz52\" (UniqueName: \"kubernetes.io/projected/951af526-1d75-4b53-8e7f-d11210ffadd1-kube-api-access-lmz52\") pod \"payload-processing-e2e-ipp-a-8e3381-5df69964cc-c9r7x\" (UID: \"951af526-1d75-4b53-8e7f-d11210ffadd1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-5df69964cc-c9r7x" Nov 01 23:04:24.896479 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:24.896423 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgbhg\" (UniqueName: \"kubernetes.io/projected/2ec00952-a3a1-436f-935f-d26957dda8fe-kube-api-access-rgbhg\") pod \"payload-pre-processing-e2e-ipp-a-8e3381-648c48588c-bf8mt\" (UID: \"2ec00952-a3a1-436f-935f-d26957dda8fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-648c48588c-bf8mt" Nov 01 23:04:25.063824 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:25.063725 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-648c48588c-bf8mt" Nov 01 23:04:25.124742 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:25.124713 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-5df69964cc-c9r7x" Nov 01 23:04:25.382875 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:04:25.381355 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ec00952_a3a1_436f_935f_d26957dda8fe.slice/crio-92f62e5941b550f3a17c4a73373a4763474e785177b644a32a7f4159c933c3cf WatchSource:0}: Error finding container 92f62e5941b550f3a17c4a73373a4763474e785177b644a32a7f4159c933c3cf: Status 404 returned error can't find the container with id 92f62e5941b550f3a17c4a73373a4763474e785177b644a32a7f4159c933c3cf Nov 01 23:04:25.389352 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:25.389327 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-648c48588c-bf8mt"] Nov 01 23:04:25.483085 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:04:25.483055 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod951af526_1d75_4b53_8e7f_d11210ffadd1.slice/crio-ca9b1078f75a601ae343fa910746a58c249732e0c5095898190aec2802adfd10 WatchSource:0}: Error finding container ca9b1078f75a601ae343fa910746a58c249732e0c5095898190aec2802adfd10: Status 404 returned error can't find the container with id ca9b1078f75a601ae343fa910746a58c249732e0c5095898190aec2802adfd10 Nov 01 23:04:25.486769 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:25.486741 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e3381-5df69964cc-c9r7x"] Nov 01 23:04:25.490484 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:25.490460 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-648c48588c-bf8mt" event={"ID":"2ec00952-a3a1-436f-935f-d26957dda8fe","Type":"ContainerStarted","Data":"0f721e68026731f516e1add2e3077d144ae4543a65ba259e342644a769a4ffd9"} Nov 01 23:04:25.490585 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:25.490510 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-648c48588c-bf8mt" event={"ID":"2ec00952-a3a1-436f-935f-d26957dda8fe","Type":"ContainerStarted","Data":"92f62e5941b550f3a17c4a73373a4763474e785177b644a32a7f4159c933c3cf"} Nov 01 23:04:25.490585 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:25.490543 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-648c48588c-bf8mt" Nov 01 23:04:25.535556 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:25.535502 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-648c48588c-bf8mt" podStartSLOduration=1.535482311 podStartE2EDuration="1.535482311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:04:25.534001298 +0000 UTC m=+2433.938591930" watchObservedRunningTime="2026-11-01 23:04:25.535482311 +0000 UTC m=+2433.940072943" Nov 01 23:04:26.521586 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:26.521536 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-5df69964cc-c9r7x" event={"ID":"951af526-1d75-4b53-8e7f-d11210ffadd1","Type":"ContainerStarted","Data":"937cc9fb2867aeeb90228374d1056a143b5cda4f0f880080f549bc5d1fce32f9"} Nov 01 23:04:26.522081 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:26.521605 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-5df69964cc-c9r7x" Nov 01 23:04:26.522081 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:26.521626 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-5df69964cc-c9r7x" event={"ID":"951af526-1d75-4b53-8e7f-d11210ffadd1","Type":"ContainerStarted","Data":"ca9b1078f75a601ae343fa910746a58c249732e0c5095898190aec2802adfd10"} Nov 01 23:04:26.540478 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:26.540429 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-5df69964cc-c9r7x" podStartSLOduration=2.540415108 podStartE2EDuration="2.540415108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:04:26.538611146 +0000 UTC m=+2434.943201777" watchObservedRunningTime="2026-11-01 23:04:26.540415108 +0000 UTC m=+2434.945005739" Nov 01 23:04:28.500006 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:28.499971 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-6dc87574c-bhqsg"] Nov 01 23:04:33.501368 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:33.501277 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-6dc87574c-bhqsg" podUID="9510c1fd-1773-4783-b86a-224f7c96c35c" containerName="payload-pre-processing" containerID="cri-o://41e1ec004046c51ecc679f63279ee6bffb5593e23ee95d86ddb5f49144079b41" gracePeriod=25 Nov 01 23:04:33.806123 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:33.806089 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-6dc87574c-bhqsg" Nov 01 23:04:33.976000 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:33.975953 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9510c1fd-1773-4783-b86a-224f7c96c35c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9510c1fd-1773-4783-b86a-224f7c96c35c-plugins-config-volume\") pod \"9510c1fd-1773-4783-b86a-224f7c96c35c\" (UID: \"9510c1fd-1773-4783-b86a-224f7c96c35c\") " Nov 01 23:04:33.977366 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:33.976664 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9510c1fd-1773-4783-b86a-224f7c96c35c-plugins-config-volume" pod "9510c1fd-1773-4783-b86a-224f7c96c35c" (UID: "9510c1fd-1773-4783-b86a-224f7c96c35c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:04:33.977644 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:33.977625 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9510c1fd-1773-4783-b86a-224f7c96c35c-kube-api-access-wfhnx\" (UniqueName: \"kubernetes.io/projected/9510c1fd-1773-4783-b86a-224f7c96c35c-kube-api-access-wfhnx\") pod \"9510c1fd-1773-4783-b86a-224f7c96c35c\" (UID: \"9510c1fd-1773-4783-b86a-224f7c96c35c\") " Nov 01 23:04:33.979076 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:33.979033 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9510c1fd-1773-4783-b86a-224f7c96c35c-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:34.011019 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:33.992561 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9510c1fd-1773-4783-b86a-224f7c96c35c-kube-api-access-wfhnx" pod "9510c1fd-1773-4783-b86a-224f7c96c35c" (UID: "9510c1fd-1773-4783-b86a-224f7c96c35c"). InnerVolumeSpecName "kube-api-access-wfhnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:04:34.081103 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:34.081019 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfhnx\" (UniqueName: \"kubernetes.io/projected/9510c1fd-1773-4783-b86a-224f7c96c35c-kube-api-access-wfhnx\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:34.606577 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:34.606534 5011 generic.go:356] "Generic (PLEG): container finished" podID="9510c1fd-1773-4783-b86a-224f7c96c35c" containerID="41e1ec004046c51ecc679f63279ee6bffb5593e23ee95d86ddb5f49144079b41" exitCode=0 Nov 01 23:04:34.607112 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:34.606585 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-6dc87574c-bhqsg" Nov 01 23:04:34.607112 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:34.606622 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-6dc87574c-bhqsg" event={"ID":"9510c1fd-1773-4783-b86a-224f7c96c35c","Type":"ContainerDied","Data":"41e1ec004046c51ecc679f63279ee6bffb5593e23ee95d86ddb5f49144079b41"} Nov 01 23:04:34.607112 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:34.606674 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-6dc87574c-bhqsg" event={"ID":"9510c1fd-1773-4783-b86a-224f7c96c35c","Type":"ContainerDied","Data":"9a72f55b800af012550051d653bbef8250bf91f569609275c50d20ff347dd20c"} Nov 01 23:04:34.607112 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:34.606697 5011 scope.go:122] "RemoveContainer" containerID="41e1ec004046c51ecc679f63279ee6bffb5593e23ee95d86ddb5f49144079b41" Nov 01 23:04:34.618725 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:34.618701 5011 scope.go:122] "RemoveContainer" containerID="41e1ec004046c51ecc679f63279ee6bffb5593e23ee95d86ddb5f49144079b41" Nov 01 23:04:34.619053 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:04:34.619033 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41e1ec004046c51ecc679f63279ee6bffb5593e23ee95d86ddb5f49144079b41\": container with ID starting with 41e1ec004046c51ecc679f63279ee6bffb5593e23ee95d86ddb5f49144079b41 not found: ID does not exist" containerID="41e1ec004046c51ecc679f63279ee6bffb5593e23ee95d86ddb5f49144079b41" Nov 01 23:04:34.619127 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:34.619065 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41e1ec004046c51ecc679f63279ee6bffb5593e23ee95d86ddb5f49144079b41"} err="failed to get container status \"41e1ec004046c51ecc679f63279ee6bffb5593e23ee95d86ddb5f49144079b41\": rpc error: code = NotFound desc = could not find container \"41e1ec004046c51ecc679f63279ee6bffb5593e23ee95d86ddb5f49144079b41\": container with ID starting with 41e1ec004046c51ecc679f63279ee6bffb5593e23ee95d86ddb5f49144079b41 not found: ID does not exist" Nov 01 23:04:34.636680 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:34.636638 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-6dc87574c-bhqsg"] Nov 01 23:04:34.640492 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:34.640463 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-6dc87574c-bhqsg"] Nov 01 23:04:35.749252 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:35.749217 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9510c1fd-1773-4783-b86a-224f7c96c35c" path="/var/lib/kubelet/pods/9510c1fd-1773-4783-b86a-224f7c96c35c/volumes" Nov 01 23:04:36.525120 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:36.525085 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-648c48588c-bf8mt" Nov 01 23:04:37.557568 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:04:37.557534 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-5df69964cc-c9r7x" Nov 01 23:04:46.988238 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:04:46.988193 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:05:02.101997 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.101960 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c"] Nov 01 23:05:02.102869 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.102846 5011 cpu_manager.go:414] "containerMap: removing container" podUID="9510c1fd-1773-4783-b86a-224f7c96c35c" containerName="payload-pre-processing" Nov 01 23:05:02.103014 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.102870 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9510c1fd-1773-4783-b86a-224f7c96c35c" containerName="payload-pre-processing" Nov 01 23:05:02.103118 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.103103 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="9510c1fd-1773-4783-b86a-224f7c96c35c" containerName="payload-pre-processing" Nov 01 23:05:02.117418 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.117384 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c" Nov 01 23:05:02.121566 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.121531 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c"] Nov 01 23:05:02.124891 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.124817 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-337dee\"/\"kube-root-ca.crt\"" Nov 01 23:05:02.125052 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.125031 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-337dee\"/\"openshift-service-ca.crt\"" Nov 01 23:05:02.125159 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.125143 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-337dee\"/\"default-dockercfg-c4h9z\"" Nov 01 23:05:02.125264 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.125224 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-337dee\"/\"ipp-route-337dee-kserve-self-signed-certs\"" Nov 01 23:05:02.306729 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.300640 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f002a7d-fa04-46d4-a4ca-4df428b7e180-tls-certs\") pod \"ipp-route-337dee-kserve-658574fc9c-bbt6c\" (UID: \"6f002a7d-fa04-46d4-a4ca-4df428b7e180\") " pod="ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c" Nov 01 23:05:02.306729 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.300691 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f002a7d-fa04-46d4-a4ca-4df428b7e180-home\") pod \"ipp-route-337dee-kserve-658574fc9c-bbt6c\" (UID: \"6f002a7d-fa04-46d4-a4ca-4df428b7e180\") " pod="ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c" Nov 01 23:05:02.306729 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.300851 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f002a7d-fa04-46d4-a4ca-4df428b7e180-tmp-dir\") pod \"ipp-route-337dee-kserve-658574fc9c-bbt6c\" (UID: \"6f002a7d-fa04-46d4-a4ca-4df428b7e180\") " pod="ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c" Nov 01 23:05:02.306729 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.300989 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f002a7d-fa04-46d4-a4ca-4df428b7e180-model-cache\") pod \"ipp-route-337dee-kserve-658574fc9c-bbt6c\" (UID: \"6f002a7d-fa04-46d4-a4ca-4df428b7e180\") " pod="ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c" Nov 01 23:05:02.306729 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.301085 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs65d\" (UniqueName: \"kubernetes.io/projected/6f002a7d-fa04-46d4-a4ca-4df428b7e180-kube-api-access-zs65d\") pod \"ipp-route-337dee-kserve-658574fc9c-bbt6c\" (UID: \"6f002a7d-fa04-46d4-a4ca-4df428b7e180\") " pod="ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c" Nov 01 23:05:02.306729 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.301115 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f002a7d-fa04-46d4-a4ca-4df428b7e180-dshm\") pod \"ipp-route-337dee-kserve-658574fc9c-bbt6c\" (UID: \"6f002a7d-fa04-46d4-a4ca-4df428b7e180\") " pod="ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c" Nov 01 23:05:02.403559 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.403458 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f002a7d-fa04-46d4-a4ca-4df428b7e180-model-cache\") pod \"ipp-route-337dee-kserve-658574fc9c-bbt6c\" (UID: \"6f002a7d-fa04-46d4-a4ca-4df428b7e180\") " pod="ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c" Nov 01 23:05:02.403744 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.403625 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs65d\" (UniqueName: \"kubernetes.io/projected/6f002a7d-fa04-46d4-a4ca-4df428b7e180-kube-api-access-zs65d\") pod \"ipp-route-337dee-kserve-658574fc9c-bbt6c\" (UID: \"6f002a7d-fa04-46d4-a4ca-4df428b7e180\") " pod="ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c" Nov 01 23:05:02.403744 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.403661 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f002a7d-fa04-46d4-a4ca-4df428b7e180-dshm\") pod \"ipp-route-337dee-kserve-658574fc9c-bbt6c\" (UID: \"6f002a7d-fa04-46d4-a4ca-4df428b7e180\") " pod="ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c" Nov 01 23:05:02.403923 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.403808 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f002a7d-fa04-46d4-a4ca-4df428b7e180-tls-certs\") pod \"ipp-route-337dee-kserve-658574fc9c-bbt6c\" (UID: \"6f002a7d-fa04-46d4-a4ca-4df428b7e180\") " pod="ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c" Nov 01 23:05:02.403923 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.403847 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f002a7d-fa04-46d4-a4ca-4df428b7e180-home\") pod \"ipp-route-337dee-kserve-658574fc9c-bbt6c\" (UID: \"6f002a7d-fa04-46d4-a4ca-4df428b7e180\") " pod="ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c" Nov 01 23:05:02.405435 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.404031 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f002a7d-fa04-46d4-a4ca-4df428b7e180-tmp-dir\") pod \"ipp-route-337dee-kserve-658574fc9c-bbt6c\" (UID: \"6f002a7d-fa04-46d4-a4ca-4df428b7e180\") " pod="ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c" Nov 01 23:05:02.405822 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.405779 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f002a7d-fa04-46d4-a4ca-4df428b7e180-model-cache\") pod \"ipp-route-337dee-kserve-658574fc9c-bbt6c\" (UID: \"6f002a7d-fa04-46d4-a4ca-4df428b7e180\") " pod="ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c" Nov 01 23:05:02.409706 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.407261 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f002a7d-fa04-46d4-a4ca-4df428b7e180-home\") pod \"ipp-route-337dee-kserve-658574fc9c-bbt6c\" (UID: \"6f002a7d-fa04-46d4-a4ca-4df428b7e180\") " pod="ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c" Nov 01 23:05:02.409706 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.407963 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f002a7d-fa04-46d4-a4ca-4df428b7e180-tmp-dir\") pod \"ipp-route-337dee-kserve-658574fc9c-bbt6c\" (UID: \"6f002a7d-fa04-46d4-a4ca-4df428b7e180\") " pod="ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c" Nov 01 23:05:02.418151 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.418114 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f002a7d-fa04-46d4-a4ca-4df428b7e180-dshm\") pod \"ipp-route-337dee-kserve-658574fc9c-bbt6c\" (UID: \"6f002a7d-fa04-46d4-a4ca-4df428b7e180\") " pod="ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c" Nov 01 23:05:02.419062 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.419010 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs65d\" (UniqueName: \"kubernetes.io/projected/6f002a7d-fa04-46d4-a4ca-4df428b7e180-kube-api-access-zs65d\") pod \"ipp-route-337dee-kserve-658574fc9c-bbt6c\" (UID: \"6f002a7d-fa04-46d4-a4ca-4df428b7e180\") " pod="ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c" Nov 01 23:05:02.423755 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.423726 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f002a7d-fa04-46d4-a4ca-4df428b7e180-tls-certs\") pod \"ipp-route-337dee-kserve-658574fc9c-bbt6c\" (UID: \"6f002a7d-fa04-46d4-a4ca-4df428b7e180\") " pod="ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c" Nov 01 23:05:02.445803 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.445763 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c" Nov 01 23:05:02.741724 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.741687 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c"] Nov 01 23:05:02.951470 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.951433 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c" event={"ID":"6f002a7d-fa04-46d4-a4ca-4df428b7e180","Type":"ContainerStarted","Data":"463bb06caca02cfc34c951360d13e0781c74e8479d3e23af9c88f57ae091d9dc"} Nov 01 23:05:02.951785 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.951480 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c" event={"ID":"6f002a7d-fa04-46d4-a4ca-4df428b7e180","Type":"ContainerStarted","Data":"6b78a1373855e42128e3468adf8062b6ff18d19976a0d17184ced89a118d6927"} Nov 01 23:05:02.973928 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:02.973880 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c" podStartSLOduration=0.973866264 podStartE2EDuration="973.866264ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:05:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:05:02.971416947 +0000 UTC m=+2471.376007580" watchObservedRunningTime="2026-11-01 23:05:02.973866264 +0000 UTC m=+2471.378456942" Nov 01 23:05:03.453855 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:03.450922 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c" Nov 01 23:05:03.682862 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:03.670432 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e3381-5df69964cc-c9r7x"] Nov 01 23:05:03.682862 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:03.677702 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-648c48588c-bf8mt"] Nov 01 23:05:08.672020 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:08.671974 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-5df69964cc-c9r7x" podUID="951af526-1d75-4b53-8e7f-d11210ffadd1" containerName="payload-processing" containerID="cri-o://937cc9fb2867aeeb90228374d1056a143b5cda4f0f880080f549bc5d1fce32f9" gracePeriod=25 Nov 01 23:05:08.679283 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:08.679218 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-648c48588c-bf8mt" podUID="2ec00952-a3a1-436f-935f-d26957dda8fe" containerName="payload-pre-processing" containerID="cri-o://0f721e68026731f516e1add2e3077d144ae4543a65ba259e342644a769a4ffd9" gracePeriod=25 Nov 01 23:05:09.065859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:09.051210 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-5df69964cc-c9r7x" Nov 01 23:05:09.081700 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:09.081665 5011 generic.go:356] "Generic (PLEG): container finished" podID="2ec00952-a3a1-436f-935f-d26957dda8fe" containerID="0f721e68026731f516e1add2e3077d144ae4543a65ba259e342644a769a4ffd9" exitCode=0 Nov 01 23:05:09.081957 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:09.081760 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-648c48588c-bf8mt" event={"ID":"2ec00952-a3a1-436f-935f-d26957dda8fe","Type":"ContainerDied","Data":"0f721e68026731f516e1add2e3077d144ae4543a65ba259e342644a769a4ffd9"} Nov 01 23:05:09.082045 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:09.082021 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-648c48588c-bf8mt" event={"ID":"2ec00952-a3a1-436f-935f-d26957dda8fe","Type":"ContainerDied","Data":"92f62e5941b550f3a17c4a73373a4763474e785177b644a32a7f4159c933c3cf"} Nov 01 23:05:09.082105 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:09.082066 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/951af526-1d75-4b53-8e7f-d11210ffadd1-kube-api-access-lmz52\" (UniqueName: \"kubernetes.io/projected/951af526-1d75-4b53-8e7f-d11210ffadd1-kube-api-access-lmz52\") pod \"951af526-1d75-4b53-8e7f-d11210ffadd1\" (UID: \"951af526-1d75-4b53-8e7f-d11210ffadd1\") " Nov 01 23:05:09.082167 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:09.082108 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/951af526-1d75-4b53-8e7f-d11210ffadd1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/951af526-1d75-4b53-8e7f-d11210ffadd1-plugins-config-volume\") pod \"951af526-1d75-4b53-8e7f-d11210ffadd1\" (UID: \"951af526-1d75-4b53-8e7f-d11210ffadd1\") " Nov 01 23:05:09.082560 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:09.082049 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92f62e5941b550f3a17c4a73373a4763474e785177b644a32a7f4159c933c3cf" Nov 01 23:05:09.085180 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:09.085136 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/951af526-1d75-4b53-8e7f-d11210ffadd1-plugins-config-volume" pod "951af526-1d75-4b53-8e7f-d11210ffadd1" (UID: "951af526-1d75-4b53-8e7f-d11210ffadd1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:05:09.087866 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:09.087493 5011 generic.go:356] "Generic (PLEG): container finished" podID="951af526-1d75-4b53-8e7f-d11210ffadd1" containerID="937cc9fb2867aeeb90228374d1056a143b5cda4f0f880080f549bc5d1fce32f9" exitCode=0 Nov 01 23:05:09.087866 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:09.087554 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-5df69964cc-c9r7x" event={"ID":"951af526-1d75-4b53-8e7f-d11210ffadd1","Type":"ContainerDied","Data":"937cc9fb2867aeeb90228374d1056a143b5cda4f0f880080f549bc5d1fce32f9"} Nov 01 23:05:09.087866 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:09.087599 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-5df69964cc-c9r7x" event={"ID":"951af526-1d75-4b53-8e7f-d11210ffadd1","Type":"ContainerDied","Data":"ca9b1078f75a601ae343fa910746a58c249732e0c5095898190aec2802adfd10"} Nov 01 23:05:09.087866 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:09.087627 5011 scope.go:122] "RemoveContainer" containerID="937cc9fb2867aeeb90228374d1056a143b5cda4f0f880080f549bc5d1fce32f9" Nov 01 23:05:09.087866 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:09.087778 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-5df69964cc-c9r7x" Nov 01 23:05:09.088278 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:09.088237 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/951af526-1d75-4b53-8e7f-d11210ffadd1-kube-api-access-lmz52" pod "951af526-1d75-4b53-8e7f-d11210ffadd1" (UID: "951af526-1d75-4b53-8e7f-d11210ffadd1"). InnerVolumeSpecName "kube-api-access-lmz52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:05:09.117137 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:09.117102 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-648c48588c-bf8mt" Nov 01 23:05:09.130858 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:09.130811 5011 scope.go:122] "RemoveContainer" containerID="937cc9fb2867aeeb90228374d1056a143b5cda4f0f880080f549bc5d1fce32f9" Nov 01 23:05:09.131253 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:05:09.131217 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"937cc9fb2867aeeb90228374d1056a143b5cda4f0f880080f549bc5d1fce32f9\": container with ID starting with 937cc9fb2867aeeb90228374d1056a143b5cda4f0f880080f549bc5d1fce32f9 not found: ID does not exist" containerID="937cc9fb2867aeeb90228374d1056a143b5cda4f0f880080f549bc5d1fce32f9" Nov 01 23:05:09.131386 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:09.131267 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"937cc9fb2867aeeb90228374d1056a143b5cda4f0f880080f549bc5d1fce32f9"} err="failed to get container status \"937cc9fb2867aeeb90228374d1056a143b5cda4f0f880080f549bc5d1fce32f9\": rpc error: code = NotFound desc = could not find container \"937cc9fb2867aeeb90228374d1056a143b5cda4f0f880080f549bc5d1fce32f9\": container with ID starting with 937cc9fb2867aeeb90228374d1056a143b5cda4f0f880080f549bc5d1fce32f9 not found: ID does not exist" Nov 01 23:05:09.187559 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:09.183659 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ec00952-a3a1-436f-935f-d26957dda8fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec00952-a3a1-436f-935f-d26957dda8fe-plugins-config-volume\") pod \"2ec00952-a3a1-436f-935f-d26957dda8fe\" (UID: \"2ec00952-a3a1-436f-935f-d26957dda8fe\") " Nov 01 23:05:09.187559 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:09.183893 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ec00952-a3a1-436f-935f-d26957dda8fe-kube-api-access-rgbhg\" (UniqueName: \"kubernetes.io/projected/2ec00952-a3a1-436f-935f-d26957dda8fe-kube-api-access-rgbhg\") pod \"2ec00952-a3a1-436f-935f-d26957dda8fe\" (UID: \"2ec00952-a3a1-436f-935f-d26957dda8fe\") " Nov 01 23:05:09.187559 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:09.184455 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/951af526-1d75-4b53-8e7f-d11210ffadd1-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:05:09.187559 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:09.184481 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmz52\" (UniqueName: \"kubernetes.io/projected/951af526-1d75-4b53-8e7f-d11210ffadd1-kube-api-access-lmz52\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:05:09.187559 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:09.187512 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ec00952-a3a1-436f-935f-d26957dda8fe-plugins-config-volume" pod "2ec00952-a3a1-436f-935f-d26957dda8fe" (UID: "2ec00952-a3a1-436f-935f-d26957dda8fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:05:09.291967 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:09.291886 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec00952-a3a1-436f-935f-d26957dda8fe-kube-api-access-rgbhg" pod "2ec00952-a3a1-436f-935f-d26957dda8fe" (UID: "2ec00952-a3a1-436f-935f-d26957dda8fe"). InnerVolumeSpecName "kube-api-access-rgbhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:05:09.291967 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:09.291891 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec00952-a3a1-436f-935f-d26957dda8fe-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:05:09.393300 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:09.393262 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgbhg\" (UniqueName: \"kubernetes.io/projected/2ec00952-a3a1-436f-935f-d26957dda8fe-kube-api-access-rgbhg\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:05:09.417653 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:09.417620 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e3381-5df69964cc-c9r7x"] Nov 01 23:05:09.423966 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:09.423936 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e3381-5df69964cc-c9r7x"] Nov 01 23:05:09.751350 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:09.751310 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="951af526-1d75-4b53-8e7f-d11210ffadd1" path="/var/lib/kubelet/pods/951af526-1d75-4b53-8e7f-d11210ffadd1/volumes" Nov 01 23:05:10.097677 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:10.097592 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-648c48588c-bf8mt" Nov 01 23:05:10.118467 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:10.118431 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-648c48588c-bf8mt"] Nov 01 23:05:10.124487 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:10.124458 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-648c48588c-bf8mt"] Nov 01 23:05:11.750382 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:11.750343 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ec00952-a3a1-436f-935f-d26957dda8fe" path="/var/lib/kubelet/pods/2ec00952-a3a1-436f-935f-d26957dda8fe/volumes" Nov 01 23:05:12.446037 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:12.445989 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c" Nov 01 23:05:12.459902 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:12.459869 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c" Nov 01 23:05:13.133391 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:13.133359 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c" Nov 01 23:05:15.149024 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:15.148980 5011 generic.go:356] "Generic (PLEG): container finished" podID="07a622eb-374c-400c-bb23-daccb27e6a3a" containerID="04f817f300e1124a616af657f8070286bb1ed67c1d73072fd5473dd7c279694b" exitCode=0 Nov 01 23:05:15.149486 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:15.149057 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" event={"ID":"07a622eb-374c-400c-bb23-daccb27e6a3a","Type":"ContainerDied","Data":"04f817f300e1124a616af657f8070286bb1ed67c1d73072fd5473dd7c279694b"} Nov 01 23:05:15.149535 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:15.149505 5011 scope.go:122] "RemoveContainer" containerID="04f817f300e1124a616af657f8070286bb1ed67c1d73072fd5473dd7c279694b" Nov 01 23:05:15.151354 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:15.151324 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:05:15.151464 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:15.151391 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:05:15.151464 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:15.151425 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:05:16.174282 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:16.174246 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" event={"ID":"07a622eb-374c-400c-bb23-daccb27e6a3a","Type":"ContainerStarted","Data":"da6cdb2dea78aa39b5d8e1a10f98d565e646c0e15aa0676e2040731895573163"} Nov 01 23:05:16.986354 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:05:16.986316 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:05:17.065103 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:17.065016 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:05:17.068544 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:17.068516 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 01 23:05:17.068689 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:17.068586 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" podUID="07a622eb-374c-400c-bb23-daccb27e6a3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 01 23:05:17.815432 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:17.815401 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:05:18.064856 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:18.064243 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 01 23:05:18.064856 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:18.064308 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" podUID="07a622eb-374c-400c-bb23-daccb27e6a3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 01 23:05:19.066615 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:19.065983 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 01 23:05:19.066615 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:19.066065 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" podUID="07a622eb-374c-400c-bb23-daccb27e6a3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 01 23:05:20.064859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:20.064215 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 01 23:05:20.064859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:20.064278 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" podUID="07a622eb-374c-400c-bb23-daccb27e6a3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 01 23:05:21.065774 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:21.065282 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 01 23:05:21.065774 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:21.065359 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" podUID="07a622eb-374c-400c-bb23-daccb27e6a3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 01 23:05:22.068233 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:22.068204 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:05:22.069159 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:22.069138 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:05:23.073544 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.073501 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg"] Nov 01 23:05:23.074696 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.074671 5011 cpu_manager.go:414] "containerMap: removing container" podUID="2ec00952-a3a1-436f-935f-d26957dda8fe" containerName="payload-pre-processing" Nov 01 23:05:23.074696 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.074694 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ec00952-a3a1-436f-935f-d26957dda8fe" containerName="payload-pre-processing" Nov 01 23:05:23.074926 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.074724 5011 cpu_manager.go:414] "containerMap: removing container" podUID="951af526-1d75-4b53-8e7f-d11210ffadd1" containerName="payload-processing" Nov 01 23:05:23.074926 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.074731 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="951af526-1d75-4b53-8e7f-d11210ffadd1" containerName="payload-processing" Nov 01 23:05:23.074926 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.074906 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ec00952-a3a1-436f-935f-d26957dda8fe" containerName="payload-pre-processing" Nov 01 23:05:23.074926 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.074923 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="951af526-1d75-4b53-8e7f-d11210ffadd1" containerName="payload-processing" Nov 01 23:05:23.079487 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.079464 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg" Nov 01 23:05:23.091164 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.091145 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-85b1ad-serving-cert\"" Nov 01 23:05:23.117730 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.117700 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg"] Nov 01 23:05:23.179846 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.179787 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd2c636a-af12-40ce-9af3-585e21d9672a-metrics-certs\") pod \"maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg\" (UID: \"dd2c636a-af12-40ce-9af3-585e21d9672a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg" Nov 01 23:05:23.180037 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.179995 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hh56\" (UniqueName: \"kubernetes.io/projected/dd2c636a-af12-40ce-9af3-585e21d9672a-kube-api-access-9hh56\") pod \"maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg\" (UID: \"dd2c636a-af12-40ce-9af3-585e21d9672a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg" Nov 01 23:05:23.180091 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.180081 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd2c636a-af12-40ce-9af3-585e21d9672a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg\" (UID: \"dd2c636a-af12-40ce-9af3-585e21d9672a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg" Nov 01 23:05:23.180146 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.180130 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd2c636a-af12-40ce-9af3-585e21d9672a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg\" (UID: \"dd2c636a-af12-40ce-9af3-585e21d9672a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg" Nov 01 23:05:23.238066 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.238029 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b"] Nov 01 23:05:23.244049 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.244005 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:23.252215 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.252168 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b"] Nov 01 23:05:23.252565 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.252547 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c00bbf-openshift-default-dockercfg-xlpqx\"" Nov 01 23:05:23.281059 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.281027 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hh56\" (UniqueName: \"kubernetes.io/projected/dd2c636a-af12-40ce-9af3-585e21d9672a-kube-api-access-9hh56\") pod \"maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg\" (UID: \"dd2c636a-af12-40ce-9af3-585e21d9672a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg" Nov 01 23:05:23.281059 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.281066 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istio-data\") pod \"e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:23.281332 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.281145 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-credential-socket\") pod \"e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:23.281332 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.281325 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-workload-certs\") pod \"e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:23.281446 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.281350 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd2c636a-af12-40ce-9af3-585e21d9672a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg\" (UID: \"dd2c636a-af12-40ce-9af3-585e21d9672a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg" Nov 01 23:05:23.281446 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.281377 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istiod-ca-cert\") pod \"e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:23.281446 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.281425 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-workload-socket\") pod \"e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:23.281612 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.281455 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istio-podinfo\") pod \"e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:23.281612 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.281475 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd2c636a-af12-40ce-9af3-585e21d9672a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg\" (UID: \"dd2c636a-af12-40ce-9af3-585e21d9672a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg" Nov 01 23:05:23.281612 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.281496 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istio-token\") pod \"e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:23.281612 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.281560 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istio-envoy\") pod \"e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:23.281612 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.281590 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mzrm\" (UniqueName: \"kubernetes.io/projected/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-kube-api-access-9mzrm\") pod \"e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:23.281949 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.281661 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd2c636a-af12-40ce-9af3-585e21d9672a-metrics-certs\") pod \"maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg\" (UID: \"dd2c636a-af12-40ce-9af3-585e21d9672a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg" Nov 01 23:05:23.282719 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.282691 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd2c636a-af12-40ce-9af3-585e21d9672a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg\" (UID: \"dd2c636a-af12-40ce-9af3-585e21d9672a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg" Nov 01 23:05:23.284491 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.284464 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd2c636a-af12-40ce-9af3-585e21d9672a-metrics-certs\") pod \"maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg\" (UID: \"dd2c636a-af12-40ce-9af3-585e21d9672a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg" Nov 01 23:05:23.293549 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.293519 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd2c636a-af12-40ce-9af3-585e21d9672a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg\" (UID: \"dd2c636a-af12-40ce-9af3-585e21d9672a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg" Nov 01 23:05:23.296606 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.296574 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hh56\" (UniqueName: \"kubernetes.io/projected/dd2c636a-af12-40ce-9af3-585e21d9672a-kube-api-access-9hh56\") pod \"maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg\" (UID: \"dd2c636a-af12-40ce-9af3-585e21d9672a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg" Nov 01 23:05:23.403096 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.384816 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istio-envoy\") pod \"e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:23.403096 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.384918 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mzrm\" (UniqueName: \"kubernetes.io/projected/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-kube-api-access-9mzrm\") pod \"e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:23.403096 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.385133 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istio-data\") pod \"e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:23.403096 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.385194 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-credential-socket\") pod \"e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:23.403096 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.385234 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-workload-certs\") pod \"e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:23.403096 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.385279 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istiod-ca-cert\") pod \"e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:23.403096 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.385309 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-workload-socket\") pod \"e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:23.403096 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.385357 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istio-podinfo\") pod \"e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:23.440386 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.403735 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istio-token\") pod \"e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:23.440386 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.408574 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istio-envoy\") pod \"e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:23.440386 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.409239 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istio-data\") pod \"e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:23.440386 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.409483 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-credential-socket\") pod \"e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:23.440386 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.409703 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-workload-certs\") pod \"e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:23.440386 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.410336 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istiod-ca-cert\") pod \"e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:23.440386 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.410616 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-workload-socket\") pod \"e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:23.440386 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.429511 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istio-podinfo\") pod \"e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:23.451792 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.451551 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istio-token\") pod \"e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:23.488879 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.456151 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg" Nov 01 23:05:23.488879 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.464508 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mzrm\" (UniqueName: \"kubernetes.io/projected/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-kube-api-access-9mzrm\") pod \"e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:23.580403 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.578229 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:23.914330 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:05:23.914242 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd2c636a_af12_40ce_9af3_585e21d9672a.slice/crio-98ee4561fa7d28609402eea06276c8597ecb12c18d239d33eca19f3676a3b513 WatchSource:0}: Error finding container 98ee4561fa7d28609402eea06276c8597ecb12c18d239d33eca19f3676a3b513: Status 404 returned error can't find the container with id 98ee4561fa7d28609402eea06276c8597ecb12c18d239d33eca19f3676a3b513 Nov 01 23:05:23.916246 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.916220 5011 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 23:05:23.918178 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.918133 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg"] Nov 01 23:05:23.922367 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:05:23.922338 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode06dee51_d25d_4e3c_bd5b_30ceb2bac53c.slice/crio-d7780b4ce41b363860e72af8f71ed10692ffe64f61bddacc180faac9a6d8b19a WatchSource:0}: Error finding container d7780b4ce41b363860e72af8f71ed10692ffe64f61bddacc180faac9a6d8b19a: Status 404 returned error can't find the container with id d7780b4ce41b363860e72af8f71ed10692ffe64f61bddacc180faac9a6d8b19a Nov 01 23:05:23.922621 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.922599 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b"] Nov 01 23:05:23.925411 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.925375 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:05:23.925512 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.925442 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:05:23.925512 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:23.925469 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:05:24.529362 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:24.529325 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" event={"ID":"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c","Type":"ContainerStarted","Data":"0a129b111f383ff4e3ff388091569fde4e1226c06fad58b5faf94d8e39b9faa7"} Nov 01 23:05:24.529362 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:24.529393 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" event={"ID":"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c","Type":"ContainerStarted","Data":"d7780b4ce41b363860e72af8f71ed10692ffe64f61bddacc180faac9a6d8b19a"} Nov 01 23:05:24.531376 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:24.531350 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg" event={"ID":"dd2c636a-af12-40ce-9af3-585e21d9672a","Type":"ContainerStarted","Data":"8e960ab1ac88f661feff2cdd7f0f3f3a9198d8bb0e7c18b95d102b1a926a498a"} Nov 01 23:05:24.531494 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:24.531396 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg" Nov 01 23:05:24.531494 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:24.531420 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg" event={"ID":"dd2c636a-af12-40ce-9af3-585e21d9672a","Type":"ContainerStarted","Data":"98ee4561fa7d28609402eea06276c8597ecb12c18d239d33eca19f3676a3b513"} Nov 01 23:05:24.549455 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:24.549405 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" podStartSLOduration=1.549392488 podStartE2EDuration="1.549392488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:05:24.548354296 +0000 UTC m=+2492.952944927" watchObservedRunningTime="2026-11-01 23:05:24.549392488 +0000 UTC m=+2492.953983120" Nov 01 23:05:24.569985 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:24.569922 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg" podStartSLOduration=1.14380552 podStartE2EDuration="1.569905346s" totalImagesPullingTime="426.099826ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:05:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:05:24.565385766 +0000 UTC m=+2492.969976400" watchObservedRunningTime="2026-11-01 23:05:24.569905346 +0000 UTC m=+2492.974495981" Nov 01 23:05:24.578760 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:24.578733 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:25.585643 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:25.585606 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:26.547867 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:26.547792 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:26.548753 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:26.548728 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:05:30.563855 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:30.554606 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg" Nov 01 23:05:37.930866 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:37.930802 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:05:40.585856 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:40.581970 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-867979c855-xlzhl"] Nov 01 23:05:40.595864 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:40.594902 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-867979c855-xlzhl" Nov 01 23:05:40.615278 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:40.615243 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-00185d\"" Nov 01 23:05:40.616883 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:40.616808 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-867979c855-xlzhl"] Nov 01 23:05:40.617096 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:40.617071 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-00185d-dockercfg-cw8b2\"" Nov 01 23:05:40.803854 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:40.802817 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc74d\" (UniqueName: \"kubernetes.io/projected/fc8d96cf-8b33-4a93-85b0-0795c3e6cd16-kube-api-access-wc74d\") pod \"payload-pre-processing-e2e-ipp-b-00185d-867979c855-xlzhl\" (UID: \"fc8d96cf-8b33-4a93-85b0-0795c3e6cd16\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-867979c855-xlzhl" Nov 01 23:05:40.803854 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:40.803068 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8d96cf-8b33-4a93-85b0-0795c3e6cd16-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-00185d-867979c855-xlzhl\" (UID: \"fc8d96cf-8b33-4a93-85b0-0795c3e6cd16\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-867979c855-xlzhl" Nov 01 23:05:40.919314 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:40.915805 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8d96cf-8b33-4a93-85b0-0795c3e6cd16-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-00185d-867979c855-xlzhl\" (UID: \"fc8d96cf-8b33-4a93-85b0-0795c3e6cd16\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-867979c855-xlzhl" Nov 01 23:05:40.919314 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:40.916102 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc74d\" (UniqueName: \"kubernetes.io/projected/fc8d96cf-8b33-4a93-85b0-0795c3e6cd16-kube-api-access-wc74d\") pod \"payload-pre-processing-e2e-ipp-b-00185d-867979c855-xlzhl\" (UID: \"fc8d96cf-8b33-4a93-85b0-0795c3e6cd16\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-867979c855-xlzhl" Nov 01 23:05:40.919314 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:40.917471 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8d96cf-8b33-4a93-85b0-0795c3e6cd16-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-00185d-867979c855-xlzhl\" (UID: \"fc8d96cf-8b33-4a93-85b0-0795c3e6cd16\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-867979c855-xlzhl" Nov 01 23:05:40.928599 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:40.928567 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc74d\" (UniqueName: \"kubernetes.io/projected/fc8d96cf-8b33-4a93-85b0-0795c3e6cd16-kube-api-access-wc74d\") pod \"payload-pre-processing-e2e-ipp-b-00185d-867979c855-xlzhl\" (UID: \"fc8d96cf-8b33-4a93-85b0-0795c3e6cd16\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-867979c855-xlzhl" Nov 01 23:05:40.934580 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:40.934551 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-867979c855-xlzhl" Nov 01 23:05:41.246196 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:41.246154 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-867979c855-xlzhl"] Nov 01 23:05:41.246343 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:05:41.246155 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8d96cf_8b33_4a93_85b0_0795c3e6cd16.slice/crio-9dacef3eb5ed42a800fece507acf91b61550bd18285fd0785e765caa6e389140 WatchSource:0}: Error finding container 9dacef3eb5ed42a800fece507acf91b61550bd18285fd0785e765caa6e389140: Status 404 returned error can't find the container with id 9dacef3eb5ed42a800fece507acf91b61550bd18285fd0785e765caa6e389140 Nov 01 23:05:41.899467 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:41.899419 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-867979c855-xlzhl" event={"ID":"fc8d96cf-8b33-4a93-85b0-0795c3e6cd16","Type":"ContainerStarted","Data":"ee4b478b5f409d190413442300839b2050d95004910f3f5dbecb712eedfe146f"} Nov 01 23:05:41.899901 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:41.899487 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-867979c855-xlzhl" event={"ID":"fc8d96cf-8b33-4a93-85b0-0795c3e6cd16","Type":"ContainerStarted","Data":"9dacef3eb5ed42a800fece507acf91b61550bd18285fd0785e765caa6e389140"} Nov 01 23:05:41.899901 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:41.899517 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-867979c855-xlzhl" Nov 01 23:05:41.918984 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:41.918925 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-867979c855-xlzhl" podStartSLOduration=1.918907031 podStartE2EDuration="1.918907031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:05:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:05:41.917170436 +0000 UTC m=+2510.321761079" watchObservedRunningTime="2026-11-01 23:05:41.918907031 +0000 UTC m=+2510.323497667" Nov 01 23:05:46.963693 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:05:46.963652 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:05:52.911170 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:05:52.911141 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-867979c855-xlzhl" Nov 01 23:06:16.961756 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:06:16.961675 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:06:25.070025 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:25.069988 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-867979c855-xlzhl"] Nov 01 23:06:30.071237 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:30.071175 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-867979c855-xlzhl" podUID="fc8d96cf-8b33-4a93-85b0-0795c3e6cd16" containerName="payload-pre-processing" containerID="cri-o://ee4b478b5f409d190413442300839b2050d95004910f3f5dbecb712eedfe146f" gracePeriod=25 Nov 01 23:06:30.369955 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:30.369922 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-867979c855-xlzhl" Nov 01 23:06:30.421399 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:30.421338 5011 generic.go:356] "Generic (PLEG): container finished" podID="fc8d96cf-8b33-4a93-85b0-0795c3e6cd16" containerID="ee4b478b5f409d190413442300839b2050d95004910f3f5dbecb712eedfe146f" exitCode=0 Nov 01 23:06:30.421757 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:30.421661 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-867979c855-xlzhl" event={"ID":"fc8d96cf-8b33-4a93-85b0-0795c3e6cd16","Type":"ContainerDied","Data":"ee4b478b5f409d190413442300839b2050d95004910f3f5dbecb712eedfe146f"} Nov 01 23:06:30.422171 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:30.422139 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-867979c855-xlzhl" event={"ID":"fc8d96cf-8b33-4a93-85b0-0795c3e6cd16","Type":"ContainerDied","Data":"9dacef3eb5ed42a800fece507acf91b61550bd18285fd0785e765caa6e389140"} Nov 01 23:06:30.422296 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:30.421678 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-867979c855-xlzhl" Nov 01 23:06:30.422296 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:30.422209 5011 scope.go:122] "RemoveContainer" containerID="ee4b478b5f409d190413442300839b2050d95004910f3f5dbecb712eedfe146f" Nov 01 23:06:30.448656 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:30.448287 5011 scope.go:122] "RemoveContainer" containerID="ee4b478b5f409d190413442300839b2050d95004910f3f5dbecb712eedfe146f" Nov 01 23:06:30.449444 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:06:30.449339 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee4b478b5f409d190413442300839b2050d95004910f3f5dbecb712eedfe146f\": container with ID starting with ee4b478b5f409d190413442300839b2050d95004910f3f5dbecb712eedfe146f not found: ID does not exist" containerID="ee4b478b5f409d190413442300839b2050d95004910f3f5dbecb712eedfe146f" Nov 01 23:06:30.449444 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:30.449382 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4b478b5f409d190413442300839b2050d95004910f3f5dbecb712eedfe146f"} err="failed to get container status \"ee4b478b5f409d190413442300839b2050d95004910f3f5dbecb712eedfe146f\": rpc error: code = NotFound desc = could not find container \"ee4b478b5f409d190413442300839b2050d95004910f3f5dbecb712eedfe146f\": container with ID starting with ee4b478b5f409d190413442300839b2050d95004910f3f5dbecb712eedfe146f not found: ID does not exist" Nov 01 23:06:30.557566 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:30.557526 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc8d96cf-8b33-4a93-85b0-0795c3e6cd16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8d96cf-8b33-4a93-85b0-0795c3e6cd16-plugins-config-volume\") pod \"fc8d96cf-8b33-4a93-85b0-0795c3e6cd16\" (UID: \"fc8d96cf-8b33-4a93-85b0-0795c3e6cd16\") " Nov 01 23:06:30.557776 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:30.557695 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc8d96cf-8b33-4a93-85b0-0795c3e6cd16-kube-api-access-wc74d\" (UniqueName: \"kubernetes.io/projected/fc8d96cf-8b33-4a93-85b0-0795c3e6cd16-kube-api-access-wc74d\") pod \"fc8d96cf-8b33-4a93-85b0-0795c3e6cd16\" (UID: \"fc8d96cf-8b33-4a93-85b0-0795c3e6cd16\") " Nov 01 23:06:30.558008 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:30.557979 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc8d96cf-8b33-4a93-85b0-0795c3e6cd16-plugins-config-volume" pod "fc8d96cf-8b33-4a93-85b0-0795c3e6cd16" (UID: "fc8d96cf-8b33-4a93-85b0-0795c3e6cd16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:06:30.558327 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:30.558308 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8d96cf-8b33-4a93-85b0-0795c3e6cd16-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:30.560035 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:30.560012 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc8d96cf-8b33-4a93-85b0-0795c3e6cd16-kube-api-access-wc74d" pod "fc8d96cf-8b33-4a93-85b0-0795c3e6cd16" (UID: "fc8d96cf-8b33-4a93-85b0-0795c3e6cd16"). InnerVolumeSpecName "kube-api-access-wc74d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:06:30.659783 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:30.659693 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc74d\" (UniqueName: \"kubernetes.io/projected/fc8d96cf-8b33-4a93-85b0-0795c3e6cd16-kube-api-access-wc74d\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:30.747868 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:30.747818 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-867979c855-xlzhl"] Nov 01 23:06:30.751859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:30.751808 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-867979c855-xlzhl"] Nov 01 23:06:31.751289 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:31.751246 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc8d96cf-8b33-4a93-85b0-0795c3e6cd16" path="/var/lib/kubelet/pods/fc8d96cf-8b33-4a93-85b0-0795c3e6cd16/volumes" Nov 01 23:06:42.589026 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:42.588975 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:06:42.589428 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:42.589076 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" podUID="e06dee51-d25d-4e3c-bd5b-30ceb2bac53c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:06:45.677730 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:45.677692 5011 generic.go:356] "Generic (PLEG): container finished" podID="e06dee51-d25d-4e3c-bd5b-30ceb2bac53c" containerID="0a129b111f383ff4e3ff388091569fde4e1226c06fad58b5faf94d8e39b9faa7" exitCode=0 Nov 01 23:06:45.678219 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:45.677774 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" event={"ID":"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c","Type":"ContainerDied","Data":"0a129b111f383ff4e3ff388091569fde4e1226c06fad58b5faf94d8e39b9faa7"} Nov 01 23:06:45.678327 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:45.678309 5011 scope.go:122] "RemoveContainer" containerID="0a129b111f383ff4e3ff388091569fde4e1226c06fad58b5faf94d8e39b9faa7" Nov 01 23:06:45.680648 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:45.680616 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:06:45.680791 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:45.680708 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:06:45.680791 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:45.680758 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:06:46.696249 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:46.696214 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" event={"ID":"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c","Type":"ContainerStarted","Data":"4beabb43d847c1aa1aa3b68e330073fe0bca140ab287394e174f06a4b99bc58b"} Nov 01 23:06:46.983923 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:06:46.983807 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:06:47.586371 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:47.586303 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:06:47.586684 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:47.586453 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Nov 01 23:06:47.586684 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:47.586527 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" podUID="e06dee51-d25d-4e3c-bd5b-30ceb2bac53c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Nov 01 23:06:48.578923 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:48.578891 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Nov 01 23:06:48.579407 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:48.578957 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" podUID="e06dee51-d25d-4e3c-bd5b-30ceb2bac53c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Nov 01 23:06:49.580459 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:49.580421 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Nov 01 23:06:49.580898 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:49.580497 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" podUID="e06dee51-d25d-4e3c-bd5b-30ceb2bac53c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Nov 01 23:06:50.075853 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.075807 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5"] Nov 01 23:06:50.076937 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.076910 5011 cpu_manager.go:414] "containerMap: removing container" podUID="fc8d96cf-8b33-4a93-85b0-0795c3e6cd16" containerName="payload-pre-processing" Nov 01 23:06:50.077071 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.077060 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc8d96cf-8b33-4a93-85b0-0795c3e6cd16" containerName="payload-pre-processing" Nov 01 23:06:50.077302 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.077292 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc8d96cf-8b33-4a93-85b0-0795c3e6cd16" containerName="payload-pre-processing" Nov 01 23:06:50.081255 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.081231 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:50.104503 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.104471 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5"] Nov 01 23:06:50.119857 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.116359 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-9d2a4e-openshift-default-dockercfg-6kq29\"" Nov 01 23:06:50.277320 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.276083 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istio-data\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:50.277320 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.276141 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istio-token\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:50.277320 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.276168 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9s2w\" (UniqueName: \"kubernetes.io/projected/bc7cb565-6bf1-400f-9a3d-f062af8270bd-kube-api-access-v9s2w\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:50.277320 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.276246 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istio-envoy\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:50.277320 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.276301 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-credential-socket\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:50.277320 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.276464 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istiod-ca-cert\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:50.277320 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.276501 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-workload-certs\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:50.277320 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.276564 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-workload-socket\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:50.277320 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.276588 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istio-podinfo\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:50.458256 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.458220 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istiod-ca-cert\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:50.458539 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.458507 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-workload-certs\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:50.458705 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.458691 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-workload-socket\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:50.458822 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.458806 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istio-podinfo\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:50.459144 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.459124 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istio-data\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:50.459263 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.459249 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istio-token\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:50.459371 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.459355 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9s2w\" (UniqueName: \"kubernetes.io/projected/bc7cb565-6bf1-400f-9a3d-f062af8270bd-kube-api-access-v9s2w\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:50.459612 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.459594 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istio-envoy\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:50.459769 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.459753 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-credential-socket\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:50.460459 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.460438 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-credential-socket\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:50.461266 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.461244 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istiod-ca-cert\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:50.473128 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.462147 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istio-data\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:50.473351 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.462367 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-workload-certs\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:50.473456 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.462587 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-workload-socket\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:50.489974 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.489901 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9s2w\" (UniqueName: \"kubernetes.io/projected/bc7cb565-6bf1-400f-9a3d-f062af8270bd-kube-api-access-v9s2w\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:50.494072 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.494009 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istio-token\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:50.501682 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.501622 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istio-podinfo\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:50.502751 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.502709 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istio-envoy\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:50.580325 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.578888 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Nov 01 23:06:50.580325 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.578957 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" podUID="e06dee51-d25d-4e3c-bd5b-30ceb2bac53c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Nov 01 23:06:50.696466 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:50.696409 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:51.019810 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:51.019782 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5"] Nov 01 23:06:51.029927 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:06:51.029893 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7cb565_6bf1_400f_9a3d_f062af8270bd.slice/crio-a1cb1c4195aca9df4cb31586d0b860f1b4e6cbfed96924ed9464369c9e40847c WatchSource:0}: Error finding container a1cb1c4195aca9df4cb31586d0b860f1b4e6cbfed96924ed9464369c9e40847c: Status 404 returned error can't find the container with id a1cb1c4195aca9df4cb31586d0b860f1b4e6cbfed96924ed9464369c9e40847c Nov 01 23:06:51.033768 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:51.033729 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:06:51.033905 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:51.033803 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:06:51.033905 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:51.033869 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:06:51.579371 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:51.579320 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Nov 01 23:06:51.579575 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:51.579429 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" podUID="e06dee51-d25d-4e3c-bd5b-30ceb2bac53c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Nov 01 23:06:51.786939 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:51.786904 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" event={"ID":"bc7cb565-6bf1-400f-9a3d-f062af8270bd","Type":"ContainerStarted","Data":"ffbe0a2f2c8528da3bf0856546f4e257f4225f6f4875da3182c647330c97d2dc"} Nov 01 23:06:51.787431 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:51.786949 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" event={"ID":"bc7cb565-6bf1-400f-9a3d-f062af8270bd","Type":"ContainerStarted","Data":"a1cb1c4195aca9df4cb31586d0b860f1b4e6cbfed96924ed9464369c9e40847c"} Nov 01 23:06:51.838786 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:51.838717 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" podStartSLOduration=2.83869769 podStartE2EDuration="2.83869769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:06:51.834264785 +0000 UTC m=+2580.238855420" watchObservedRunningTime="2026-11-01 23:06:51.83869769 +0000 UTC m=+2580.243288326" Nov 01 23:06:52.703366 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:52.703217 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:52.737253 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:52.737222 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:52.790654 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:52.790626 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:52.792346 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:52.792309 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:06:52.807998 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:52.807971 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:06:52.808660 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:52.808639 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:06:52.809609 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:52.809589 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:06:55.378567 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:55.378469 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-6d69769b69-xzsjh"] Nov 01 23:06:55.385027 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:55.385002 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-6d69769b69-xzsjh" Nov 01 23:06:55.391771 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:55.391721 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-6d69769b69-xzsjh"] Nov 01 23:06:55.394093 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:55.394068 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9d2a4e-dockercfg-bhzlj\"" Nov 01 23:06:55.394301 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:55.394172 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9d2a4e\"" Nov 01 23:06:55.396945 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:55.396915 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-6bcf5b98c9-2qh8l"] Nov 01 23:06:55.402025 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:55.401999 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-6bcf5b98c9-2qh8l" Nov 01 23:06:55.416891 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:55.416868 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-6bcf5b98c9-2qh8l"] Nov 01 23:06:55.456101 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:55.456060 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8da83a6b-b3fb-4974-8edc-d225c5c56f93-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9d2a4e-6d69769b69-xzsjh\" (UID: \"8da83a6b-b3fb-4974-8edc-d225c5c56f93\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-6d69769b69-xzsjh" Nov 01 23:06:55.456296 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:55.456178 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/072ed749-d3d4-423d-a6e5-4c3961649970-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9d2a4e-6bcf5b98c9-2qh8l\" (UID: \"072ed749-d3d4-423d-a6e5-4c3961649970\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-6bcf5b98c9-2qh8l" Nov 01 23:06:55.456296 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:55.456239 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pnr8\" (UniqueName: \"kubernetes.io/projected/8da83a6b-b3fb-4974-8edc-d225c5c56f93-kube-api-access-7pnr8\") pod \"payload-processing-e2e-ipp-b-9d2a4e-6d69769b69-xzsjh\" (UID: \"8da83a6b-b3fb-4974-8edc-d225c5c56f93\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-6d69769b69-xzsjh" Nov 01 23:06:55.456438 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:55.456421 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hck57\" (UniqueName: \"kubernetes.io/projected/072ed749-d3d4-423d-a6e5-4c3961649970-kube-api-access-hck57\") pod \"payload-pre-processing-e2e-ipp-b-9d2a4e-6bcf5b98c9-2qh8l\" (UID: \"072ed749-d3d4-423d-a6e5-4c3961649970\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-6bcf5b98c9-2qh8l" Nov 01 23:06:55.557737 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:55.557696 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8da83a6b-b3fb-4974-8edc-d225c5c56f93-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9d2a4e-6d69769b69-xzsjh\" (UID: \"8da83a6b-b3fb-4974-8edc-d225c5c56f93\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-6d69769b69-xzsjh" Nov 01 23:06:55.557948 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:55.557897 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/072ed749-d3d4-423d-a6e5-4c3961649970-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9d2a4e-6bcf5b98c9-2qh8l\" (UID: \"072ed749-d3d4-423d-a6e5-4c3961649970\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-6bcf5b98c9-2qh8l" Nov 01 23:06:55.558031 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:55.557965 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pnr8\" (UniqueName: \"kubernetes.io/projected/8da83a6b-b3fb-4974-8edc-d225c5c56f93-kube-api-access-7pnr8\") pod \"payload-processing-e2e-ipp-b-9d2a4e-6d69769b69-xzsjh\" (UID: \"8da83a6b-b3fb-4974-8edc-d225c5c56f93\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-6d69769b69-xzsjh" Nov 01 23:06:55.558181 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:55.558163 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hck57\" (UniqueName: \"kubernetes.io/projected/072ed749-d3d4-423d-a6e5-4c3961649970-kube-api-access-hck57\") pod \"payload-pre-processing-e2e-ipp-b-9d2a4e-6bcf5b98c9-2qh8l\" (UID: \"072ed749-d3d4-423d-a6e5-4c3961649970\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-6bcf5b98c9-2qh8l" Nov 01 23:06:55.558687 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:55.558658 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8da83a6b-b3fb-4974-8edc-d225c5c56f93-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9d2a4e-6d69769b69-xzsjh\" (UID: \"8da83a6b-b3fb-4974-8edc-d225c5c56f93\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-6d69769b69-xzsjh" Nov 01 23:06:55.558792 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:55.558693 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/072ed749-d3d4-423d-a6e5-4c3961649970-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9d2a4e-6bcf5b98c9-2qh8l\" (UID: \"072ed749-d3d4-423d-a6e5-4c3961649970\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-6bcf5b98c9-2qh8l" Nov 01 23:06:55.569246 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:55.569210 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hck57\" (UniqueName: \"kubernetes.io/projected/072ed749-d3d4-423d-a6e5-4c3961649970-kube-api-access-hck57\") pod \"payload-pre-processing-e2e-ipp-b-9d2a4e-6bcf5b98c9-2qh8l\" (UID: \"072ed749-d3d4-423d-a6e5-4c3961649970\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-6bcf5b98c9-2qh8l" Nov 01 23:06:55.569626 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:55.569601 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pnr8\" (UniqueName: \"kubernetes.io/projected/8da83a6b-b3fb-4974-8edc-d225c5c56f93-kube-api-access-7pnr8\") pod \"payload-processing-e2e-ipp-b-9d2a4e-6d69769b69-xzsjh\" (UID: \"8da83a6b-b3fb-4974-8edc-d225c5c56f93\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-6d69769b69-xzsjh" Nov 01 23:06:55.705596 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:55.705558 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-6d69769b69-xzsjh" Nov 01 23:06:55.726554 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:55.726508 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-6bcf5b98c9-2qh8l" Nov 01 23:06:56.215270 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:56.205662 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-6d69769b69-xzsjh"] Nov 01 23:06:56.215270 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:06:56.207527 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8da83a6b_b3fb_4974_8edc_d225c5c56f93.slice/crio-f4c07516ea4cecb467613cd9997b29df3a1bc2a869b4b1eb17e5c292784f8a23 WatchSource:0}: Error finding container f4c07516ea4cecb467613cd9997b29df3a1bc2a869b4b1eb17e5c292784f8a23: Status 404 returned error can't find the container with id f4c07516ea4cecb467613cd9997b29df3a1bc2a869b4b1eb17e5c292784f8a23 Nov 01 23:06:56.233899 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:56.233859 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-6bcf5b98c9-2qh8l"] Nov 01 23:06:56.250065 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:06:56.250031 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod072ed749_d3d4_423d_a6e5_4c3961649970.slice/crio-765db6d7af79bd6eb051c817ed2e6c5f7bc0c1481dde845e738a86c8b443f513 WatchSource:0}: Error finding container 765db6d7af79bd6eb051c817ed2e6c5f7bc0c1481dde845e738a86c8b443f513: Status 404 returned error can't find the container with id 765db6d7af79bd6eb051c817ed2e6c5f7bc0c1481dde845e738a86c8b443f513 Nov 01 23:06:56.843361 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:56.843317 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-6bcf5b98c9-2qh8l" event={"ID":"072ed749-d3d4-423d-a6e5-4c3961649970","Type":"ContainerStarted","Data":"f3d0ab7af89e2dce19607196112e40d0b086efc11f2e3cffefa7080e79ac033e"} Nov 01 23:06:56.843930 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:56.843389 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-6bcf5b98c9-2qh8l" event={"ID":"072ed749-d3d4-423d-a6e5-4c3961649970","Type":"ContainerStarted","Data":"765db6d7af79bd6eb051c817ed2e6c5f7bc0c1481dde845e738a86c8b443f513"} Nov 01 23:06:56.843930 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:56.843496 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-6bcf5b98c9-2qh8l" Nov 01 23:06:56.845329 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:56.845302 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-6d69769b69-xzsjh" event={"ID":"8da83a6b-b3fb-4974-8edc-d225c5c56f93","Type":"ContainerStarted","Data":"7e327a9f881847c57c340be38d581edc468c8a559bb8313d9581f2ab7dcfe41f"} Nov 01 23:06:56.845450 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:56.845355 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-6d69769b69-xzsjh" event={"ID":"8da83a6b-b3fb-4974-8edc-d225c5c56f93","Type":"ContainerStarted","Data":"f4c07516ea4cecb467613cd9997b29df3a1bc2a869b4b1eb17e5c292784f8a23"} Nov 01 23:06:56.845450 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:56.845425 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-6d69769b69-xzsjh" Nov 01 23:06:56.864672 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:56.864623 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-6bcf5b98c9-2qh8l" podStartSLOduration=1.8646085289999998 podStartE2EDuration="1.864608529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:06:56.859981474 +0000 UTC m=+2585.264572106" watchObservedRunningTime="2026-11-01 23:06:56.864608529 +0000 UTC m=+2585.269199196" Nov 01 23:06:56.877513 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:06:56.877452 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-6d69769b69-xzsjh" podStartSLOduration=1.8774374360000001 podStartE2EDuration="1.877437436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:06:56.876243415 +0000 UTC m=+2585.280834050" watchObservedRunningTime="2026-11-01 23:06:56.877437436 +0000 UTC m=+2585.282028076" Nov 01 23:07:07.858812 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:07.858778 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-6d69769b69-xzsjh" Nov 01 23:07:07.859300 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:07.858917 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-6bcf5b98c9-2qh8l" Nov 01 23:07:12.535583 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:12.535547 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg"] Nov 01 23:07:12.536314 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:12.536284 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg" podUID="dd2c636a-af12-40ce-9af3-585e21d9672a" containerName="maas-api" containerID="cri-o://8e960ab1ac88f661feff2cdd7f0f3f3a9198d8bb0e7c18b95d102b1a926a498a" gracePeriod=30 Nov 01 23:07:13.135276 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:13.135234 5011 generic.go:356] "Generic (PLEG): container finished" podID="dd2c636a-af12-40ce-9af3-585e21d9672a" containerID="8e960ab1ac88f661feff2cdd7f0f3f3a9198d8bb0e7c18b95d102b1a926a498a" exitCode=0 Nov 01 23:07:13.135474 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:13.135277 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg" event={"ID":"dd2c636a-af12-40ce-9af3-585e21d9672a","Type":"ContainerDied","Data":"8e960ab1ac88f661feff2cdd7f0f3f3a9198d8bb0e7c18b95d102b1a926a498a"} Nov 01 23:07:13.135474 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:13.135391 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg" event={"ID":"dd2c636a-af12-40ce-9af3-585e21d9672a","Type":"ContainerDied","Data":"98ee4561fa7d28609402eea06276c8597ecb12c18d239d33eca19f3676a3b513"} Nov 01 23:07:13.135474 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:13.135419 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98ee4561fa7d28609402eea06276c8597ecb12c18d239d33eca19f3676a3b513" Nov 01 23:07:13.137706 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:13.137688 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg" Nov 01 23:07:13.320938 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:13.318231 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd2c636a-af12-40ce-9af3-585e21d9672a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd2c636a-af12-40ce-9af3-585e21d9672a-maas-api-tls\") pod \"dd2c636a-af12-40ce-9af3-585e21d9672a\" (UID: \"dd2c636a-af12-40ce-9af3-585e21d9672a\") " Nov 01 23:07:13.320938 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:13.318299 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd2c636a-af12-40ce-9af3-585e21d9672a-kube-api-access-9hh56\" (UniqueName: \"kubernetes.io/projected/dd2c636a-af12-40ce-9af3-585e21d9672a-kube-api-access-9hh56\") pod \"dd2c636a-af12-40ce-9af3-585e21d9672a\" (UID: \"dd2c636a-af12-40ce-9af3-585e21d9672a\") " Nov 01 23:07:13.320938 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:13.318377 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd2c636a-af12-40ce-9af3-585e21d9672a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd2c636a-af12-40ce-9af3-585e21d9672a-metrics-certs\") pod \"dd2c636a-af12-40ce-9af3-585e21d9672a\" (UID: \"dd2c636a-af12-40ce-9af3-585e21d9672a\") " Nov 01 23:07:13.320938 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:13.318533 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd2c636a-af12-40ce-9af3-585e21d9672a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd2c636a-af12-40ce-9af3-585e21d9672a-openshift-service-ca\") pod \"dd2c636a-af12-40ce-9af3-585e21d9672a\" (UID: \"dd2c636a-af12-40ce-9af3-585e21d9672a\") " Nov 01 23:07:13.329860 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:13.328035 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd2c636a-af12-40ce-9af3-585e21d9672a-openshift-service-ca" pod "dd2c636a-af12-40ce-9af3-585e21d9672a" (UID: "dd2c636a-af12-40ce-9af3-585e21d9672a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:07:13.370290 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:13.362291 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd2c636a-af12-40ce-9af3-585e21d9672a-maas-api-tls" pod "dd2c636a-af12-40ce-9af3-585e21d9672a" (UID: "dd2c636a-af12-40ce-9af3-585e21d9672a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:07:13.380546 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:13.377534 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd2c636a-af12-40ce-9af3-585e21d9672a-metrics-certs" pod "dd2c636a-af12-40ce-9af3-585e21d9672a" (UID: "dd2c636a-af12-40ce-9af3-585e21d9672a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:07:13.380546 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:13.378524 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd2c636a-af12-40ce-9af3-585e21d9672a-kube-api-access-9hh56" pod "dd2c636a-af12-40ce-9af3-585e21d9672a" (UID: "dd2c636a-af12-40ce-9af3-585e21d9672a"). InnerVolumeSpecName "kube-api-access-9hh56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:13.420742 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:13.420710 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd2c636a-af12-40ce-9af3-585e21d9672a-openshift-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:13.420742 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:13.420742 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd2c636a-af12-40ce-9af3-585e21d9672a-maas-api-tls\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:13.421031 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:13.420755 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hh56\" (UniqueName: \"kubernetes.io/projected/dd2c636a-af12-40ce-9af3-585e21d9672a-kube-api-access-9hh56\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:13.421031 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:13.420769 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd2c636a-af12-40ce-9af3-585e21d9672a-metrics-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:14.144566 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:14.144475 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg" Nov 01 23:07:14.163910 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:14.163879 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg"] Nov 01 23:07:14.167960 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:14.167932 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-85b1ad-6df7798c7b-8fnpg"] Nov 01 23:07:15.750246 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:15.750199 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd2c636a-af12-40ce-9af3-585e21d9672a" path="/var/lib/kubelet/pods/dd2c636a-af12-40ce-9af3-585e21d9672a/volumes" Nov 01 23:07:16.961604 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:07:16.961565 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:07:30.544875 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:30.544807 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2f3705-64d47ccbdf-xgv67"] Nov 01 23:07:30.571865 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:30.568767 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2f3705-855984bb47-ztmpt"] Nov 01 23:07:32.467572 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:32.467482 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:07:35.545524 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:35.545470 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-2f3705-64d47ccbdf-xgv67" podUID="0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce" containerName="payload-processing" containerID="cri-o://e6677851f8cddef76b5db75ac3e4d1e306603fb21b97ae112df858390f784972" gracePeriod=25 Nov 01 23:07:35.569203 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:35.569160 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2f3705-855984bb47-ztmpt" podUID="c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583" containerName="payload-pre-processing" containerID="cri-o://3e841d9918502a08128a3d7cbf9ee34e3cef12fb39a1bd5651109c7d1cc9d9c9" gracePeriod=25 Nov 01 23:07:35.735929 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:35.734155 5011 generic.go:356] "Generic (PLEG): container finished" podID="0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce" containerID="e6677851f8cddef76b5db75ac3e4d1e306603fb21b97ae112df858390f784972" exitCode=0 Nov 01 23:07:35.735929 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:35.734339 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-2f3705-64d47ccbdf-xgv67" event={"ID":"0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce","Type":"ContainerDied","Data":"e6677851f8cddef76b5db75ac3e4d1e306603fb21b97ae112df858390f784972"} Nov 01 23:07:35.862198 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:35.862167 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-2f3705-64d47ccbdf-xgv67" Nov 01 23:07:35.952287 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:35.952243 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce-kube-api-access-nhwwm\" (UniqueName: \"kubernetes.io/projected/0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce-kube-api-access-nhwwm\") pod \"0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce\" (UID: \"0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce\") " Nov 01 23:07:35.952629 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:35.952610 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce-plugins-config-volume\") pod \"0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce\" (UID: \"0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce\") " Nov 01 23:07:35.952942 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:35.952916 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce-plugins-config-volume" pod "0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce" (UID: "0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:07:35.953128 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:35.953113 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:35.954736 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:35.954706 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce-kube-api-access-nhwwm" pod "0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce" (UID: "0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce"). InnerVolumeSpecName "kube-api-access-nhwwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:36.055070 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:36.055030 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhwwm\" (UniqueName: \"kubernetes.io/projected/0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce-kube-api-access-nhwwm\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:36.743375 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:36.743338 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-2f3705-64d47ccbdf-xgv67" Nov 01 23:07:36.743826 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:36.743345 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-2f3705-64d47ccbdf-xgv67" event={"ID":"0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce","Type":"ContainerDied","Data":"78ae3d827fc6d07a9b60aee15484cea0e4cc51a88e26d4dbb0021e5126f25fb2"} Nov 01 23:07:36.743826 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:36.743493 5011 scope.go:122] "RemoveContainer" containerID="e6677851f8cddef76b5db75ac3e4d1e306603fb21b97ae112df858390f784972" Nov 01 23:07:36.797169 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:36.797138 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2f3705-64d47ccbdf-xgv67"] Nov 01 23:07:36.808196 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:36.808167 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2f3705-64d47ccbdf-xgv67"] Nov 01 23:07:37.758428 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:37.758387 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce" path="/var/lib/kubelet/pods/0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce/volumes" Nov 01 23:07:39.673140 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:39.673102 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-6d69769b69-xzsjh"] Nov 01 23:07:39.688185 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:39.688152 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-6bcf5b98c9-2qh8l"] Nov 01 23:07:43.691596 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:43.691553 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2f3705-855984bb47-ztmpt" podUID="c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:07:44.674380 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:44.674340 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-6d69769b69-xzsjh" podUID="8da83a6b-b3fb-4974-8edc-d225c5c56f93" containerName="payload-processing" containerID="cri-o://7e327a9f881847c57c340be38d581edc468c8a559bb8313d9581f2ab7dcfe41f" gracePeriod=25 Nov 01 23:07:44.688882 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:44.688816 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-6bcf5b98c9-2qh8l" podUID="072ed749-d3d4-423d-a6e5-4c3961649970" containerName="payload-pre-processing" containerID="cri-o://f3d0ab7af89e2dce19607196112e40d0b086efc11f2e3cffefa7080e79ac033e" gracePeriod=25 Nov 01 23:07:44.889178 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:44.889139 5011 generic.go:356] "Generic (PLEG): container finished" podID="072ed749-d3d4-423d-a6e5-4c3961649970" containerID="f3d0ab7af89e2dce19607196112e40d0b086efc11f2e3cffefa7080e79ac033e" exitCode=0 Nov 01 23:07:44.889664 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:44.889230 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-6bcf5b98c9-2qh8l" event={"ID":"072ed749-d3d4-423d-a6e5-4c3961649970","Type":"ContainerDied","Data":"f3d0ab7af89e2dce19607196112e40d0b086efc11f2e3cffefa7080e79ac033e"} Nov 01 23:07:44.891475 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:44.891306 5011 generic.go:356] "Generic (PLEG): container finished" podID="8da83a6b-b3fb-4974-8edc-d225c5c56f93" containerID="7e327a9f881847c57c340be38d581edc468c8a559bb8313d9581f2ab7dcfe41f" exitCode=0 Nov 01 23:07:44.891475 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:44.891364 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-6d69769b69-xzsjh" event={"ID":"8da83a6b-b3fb-4974-8edc-d225c5c56f93","Type":"ContainerDied","Data":"7e327a9f881847c57c340be38d581edc468c8a559bb8313d9581f2ab7dcfe41f"} Nov 01 23:07:45.043472 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:45.043133 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-6d69769b69-xzsjh" Nov 01 23:07:45.124471 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:45.124444 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-6bcf5b98c9-2qh8l" Nov 01 23:07:45.186787 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:45.186755 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8da83a6b-b3fb-4974-8edc-d225c5c56f93-kube-api-access-7pnr8\" (UniqueName: \"kubernetes.io/projected/8da83a6b-b3fb-4974-8edc-d225c5c56f93-kube-api-access-7pnr8\") pod \"8da83a6b-b3fb-4974-8edc-d225c5c56f93\" (UID: \"8da83a6b-b3fb-4974-8edc-d225c5c56f93\") " Nov 01 23:07:45.187113 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:45.187090 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8da83a6b-b3fb-4974-8edc-d225c5c56f93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8da83a6b-b3fb-4974-8edc-d225c5c56f93-plugins-config-volume\") pod \"8da83a6b-b3fb-4974-8edc-d225c5c56f93\" (UID: \"8da83a6b-b3fb-4974-8edc-d225c5c56f93\") " Nov 01 23:07:45.187693 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:45.187635 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8da83a6b-b3fb-4974-8edc-d225c5c56f93-plugins-config-volume" pod "8da83a6b-b3fb-4974-8edc-d225c5c56f93" (UID: "8da83a6b-b3fb-4974-8edc-d225c5c56f93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:07:45.187912 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:45.187887 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8da83a6b-b3fb-4974-8edc-d225c5c56f93-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:45.189758 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:45.189732 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8da83a6b-b3fb-4974-8edc-d225c5c56f93-kube-api-access-7pnr8" pod "8da83a6b-b3fb-4974-8edc-d225c5c56f93" (UID: "8da83a6b-b3fb-4974-8edc-d225c5c56f93"). InnerVolumeSpecName "kube-api-access-7pnr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:45.289231 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:45.289144 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/072ed749-d3d4-423d-a6e5-4c3961649970-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/072ed749-d3d4-423d-a6e5-4c3961649970-plugins-config-volume\") pod \"072ed749-d3d4-423d-a6e5-4c3961649970\" (UID: \"072ed749-d3d4-423d-a6e5-4c3961649970\") " Nov 01 23:07:45.289406 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:45.289335 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/072ed749-d3d4-423d-a6e5-4c3961649970-kube-api-access-hck57\" (UniqueName: \"kubernetes.io/projected/072ed749-d3d4-423d-a6e5-4c3961649970-kube-api-access-hck57\") pod \"072ed749-d3d4-423d-a6e5-4c3961649970\" (UID: \"072ed749-d3d4-423d-a6e5-4c3961649970\") " Nov 01 23:07:45.289634 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:45.289604 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/072ed749-d3d4-423d-a6e5-4c3961649970-plugins-config-volume" pod "072ed749-d3d4-423d-a6e5-4c3961649970" (UID: "072ed749-d3d4-423d-a6e5-4c3961649970"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:07:45.290265 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:45.290238 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pnr8\" (UniqueName: \"kubernetes.io/projected/8da83a6b-b3fb-4974-8edc-d225c5c56f93-kube-api-access-7pnr8\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:45.290265 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:45.290266 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/072ed749-d3d4-423d-a6e5-4c3961649970-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:45.291781 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:45.291749 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/072ed749-d3d4-423d-a6e5-4c3961649970-kube-api-access-hck57" pod "072ed749-d3d4-423d-a6e5-4c3961649970" (UID: "072ed749-d3d4-423d-a6e5-4c3961649970"). InnerVolumeSpecName "kube-api-access-hck57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:45.391775 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:45.391743 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hck57\" (UniqueName: \"kubernetes.io/projected/072ed749-d3d4-423d-a6e5-4c3961649970-kube-api-access-hck57\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:45.901438 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:45.901411 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-6bcf5b98c9-2qh8l" Nov 01 23:07:45.901438 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:45.901417 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-6bcf5b98c9-2qh8l" event={"ID":"072ed749-d3d4-423d-a6e5-4c3961649970","Type":"ContainerDied","Data":"765db6d7af79bd6eb051c817ed2e6c5f7bc0c1481dde845e738a86c8b443f513"} Nov 01 23:07:45.901927 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:45.901474 5011 scope.go:122] "RemoveContainer" containerID="f3d0ab7af89e2dce19607196112e40d0b086efc11f2e3cffefa7080e79ac033e" Nov 01 23:07:45.903329 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:45.903303 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-6d69769b69-xzsjh" event={"ID":"8da83a6b-b3fb-4974-8edc-d225c5c56f93","Type":"ContainerDied","Data":"f4c07516ea4cecb467613cd9997b29df3a1bc2a869b4b1eb17e5c292784f8a23"} Nov 01 23:07:45.903428 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:45.903351 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-6d69769b69-xzsjh" Nov 01 23:07:45.913437 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:45.913418 5011 scope.go:122] "RemoveContainer" containerID="7e327a9f881847c57c340be38d581edc468c8a559bb8313d9581f2ab7dcfe41f" Nov 01 23:07:45.922457 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:45.922403 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-6bcf5b98c9-2qh8l"] Nov 01 23:07:45.926459 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:45.926438 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-6bcf5b98c9-2qh8l"] Nov 01 23:07:45.938336 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:45.938314 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-6d69769b69-xzsjh"] Nov 01 23:07:45.943343 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:45.943316 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-6d69769b69-xzsjh"] Nov 01 23:07:47.039942 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:07:47.031033 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:07:47.748653 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:47.748610 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="072ed749-d3d4-423d-a6e5-4c3961649970" path="/var/lib/kubelet/pods/072ed749-d3d4-423d-a6e5-4c3961649970/volumes" Nov 01 23:07:47.749169 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:47.749146 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8da83a6b-b3fb-4974-8edc-d225c5c56f93" path="/var/lib/kubelet/pods/8da83a6b-b3fb-4974-8edc-d225c5c56f93/volumes" Nov 01 23:07:53.690015 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:53.689972 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2f3705-855984bb47-ztmpt" podUID="c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:07:54.747594 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:54.747558 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-77f6b5c75-k7qln"] Nov 01 23:07:54.748354 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:54.748335 5011 cpu_manager.go:414] "containerMap: removing container" podUID="dd2c636a-af12-40ce-9af3-585e21d9672a" containerName="maas-api" Nov 01 23:07:54.748354 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:54.748355 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd2c636a-af12-40ce-9af3-585e21d9672a" containerName="maas-api" Nov 01 23:07:54.748448 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:54.748404 5011 cpu_manager.go:414] "containerMap: removing container" podUID="072ed749-d3d4-423d-a6e5-4c3961649970" containerName="payload-pre-processing" Nov 01 23:07:54.748448 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:54.748410 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="072ed749-d3d4-423d-a6e5-4c3961649970" containerName="payload-pre-processing" Nov 01 23:07:54.748448 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:54.748421 5011 cpu_manager.go:414] "containerMap: removing container" podUID="0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce" containerName="payload-processing" Nov 01 23:07:54.748448 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:54.748426 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce" containerName="payload-processing" Nov 01 23:07:54.748448 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:54.748434 5011 cpu_manager.go:414] "containerMap: removing container" podUID="8da83a6b-b3fb-4974-8edc-d225c5c56f93" containerName="payload-processing" Nov 01 23:07:54.748448 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:54.748439 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8da83a6b-b3fb-4974-8edc-d225c5c56f93" containerName="payload-processing" Nov 01 23:07:54.748680 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:54.748544 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="072ed749-d3d4-423d-a6e5-4c3961649970" containerName="payload-pre-processing" Nov 01 23:07:54.748680 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:54.748555 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e8a8d7b-bc08-4d30-a435-dfd4d4ed76ce" containerName="payload-processing" Nov 01 23:07:54.748680 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:54.748563 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="8da83a6b-b3fb-4974-8edc-d225c5c56f93" containerName="payload-processing" Nov 01 23:07:54.748680 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:54.748570 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd2c636a-af12-40ce-9af3-585e21d9672a" containerName="maas-api" Nov 01 23:07:54.753650 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:54.753630 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-77f6b5c75-k7qln" Nov 01 23:07:54.761167 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:54.761145 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-01e30d\"" Nov 01 23:07:54.761305 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:54.761273 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-01e30d-dockercfg-jtkmf\"" Nov 01 23:07:54.765183 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:54.765155 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-77f6b5c75-k7qln"] Nov 01 23:07:54.803889 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:54.803851 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-01e30d-54876bc6b7-sj6gr"] Nov 01 23:07:54.815769 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:54.815727 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmzv7\" (UniqueName: \"kubernetes.io/projected/5e9a6cde-55d0-40d6-8fe7-e54452ef8da7-kube-api-access-dmzv7\") pod \"payload-pre-processing-e2e-ipp-a-01e30d-77f6b5c75-k7qln\" (UID: \"5e9a6cde-55d0-40d6-8fe7-e54452ef8da7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-77f6b5c75-k7qln" Nov 01 23:07:54.816163 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:54.816133 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e9a6cde-55d0-40d6-8fe7-e54452ef8da7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-01e30d-77f6b5c75-k7qln\" (UID: \"5e9a6cde-55d0-40d6-8fe7-e54452ef8da7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-77f6b5c75-k7qln" Nov 01 23:07:54.817409 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:54.817388 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-54876bc6b7-sj6gr" Nov 01 23:07:54.818396 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:54.817293 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-01e30d-54876bc6b7-sj6gr"] Nov 01 23:07:55.003807 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:55.003709 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6207c82-7f15-4c9a-9012-5c1aa1659dd0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-01e30d-54876bc6b7-sj6gr\" (UID: \"a6207c82-7f15-4c9a-9012-5c1aa1659dd0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-54876bc6b7-sj6gr" Nov 01 23:07:55.122319 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:55.112317 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh9g6\" (UniqueName: \"kubernetes.io/projected/a6207c82-7f15-4c9a-9012-5c1aa1659dd0-kube-api-access-lh9g6\") pod \"payload-processing-e2e-ipp-a-01e30d-54876bc6b7-sj6gr\" (UID: \"a6207c82-7f15-4c9a-9012-5c1aa1659dd0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-54876bc6b7-sj6gr" Nov 01 23:07:55.122319 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:55.112643 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmzv7\" (UniqueName: \"kubernetes.io/projected/5e9a6cde-55d0-40d6-8fe7-e54452ef8da7-kube-api-access-dmzv7\") pod \"payload-pre-processing-e2e-ipp-a-01e30d-77f6b5c75-k7qln\" (UID: \"5e9a6cde-55d0-40d6-8fe7-e54452ef8da7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-77f6b5c75-k7qln" Nov 01 23:07:55.122319 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:55.112763 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e9a6cde-55d0-40d6-8fe7-e54452ef8da7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-01e30d-77f6b5c75-k7qln\" (UID: \"5e9a6cde-55d0-40d6-8fe7-e54452ef8da7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-77f6b5c75-k7qln" Nov 01 23:07:55.122319 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:55.113984 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e9a6cde-55d0-40d6-8fe7-e54452ef8da7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-01e30d-77f6b5c75-k7qln\" (UID: \"5e9a6cde-55d0-40d6-8fe7-e54452ef8da7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-77f6b5c75-k7qln" Nov 01 23:07:55.142310 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:55.142272 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmzv7\" (UniqueName: \"kubernetes.io/projected/5e9a6cde-55d0-40d6-8fe7-e54452ef8da7-kube-api-access-dmzv7\") pod \"payload-pre-processing-e2e-ipp-a-01e30d-77f6b5c75-k7qln\" (UID: \"5e9a6cde-55d0-40d6-8fe7-e54452ef8da7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-77f6b5c75-k7qln" Nov 01 23:07:55.214140 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:55.214105 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6207c82-7f15-4c9a-9012-5c1aa1659dd0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-01e30d-54876bc6b7-sj6gr\" (UID: \"a6207c82-7f15-4c9a-9012-5c1aa1659dd0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-54876bc6b7-sj6gr" Nov 01 23:07:55.214140 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:55.214146 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh9g6\" (UniqueName: \"kubernetes.io/projected/a6207c82-7f15-4c9a-9012-5c1aa1659dd0-kube-api-access-lh9g6\") pod \"payload-processing-e2e-ipp-a-01e30d-54876bc6b7-sj6gr\" (UID: \"a6207c82-7f15-4c9a-9012-5c1aa1659dd0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-54876bc6b7-sj6gr" Nov 01 23:07:55.214908 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:55.214883 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6207c82-7f15-4c9a-9012-5c1aa1659dd0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-01e30d-54876bc6b7-sj6gr\" (UID: \"a6207c82-7f15-4c9a-9012-5c1aa1659dd0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-54876bc6b7-sj6gr" Nov 01 23:07:55.222864 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:55.222802 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh9g6\" (UniqueName: \"kubernetes.io/projected/a6207c82-7f15-4c9a-9012-5c1aa1659dd0-kube-api-access-lh9g6\") pod \"payload-processing-e2e-ipp-a-01e30d-54876bc6b7-sj6gr\" (UID: \"a6207c82-7f15-4c9a-9012-5c1aa1659dd0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-54876bc6b7-sj6gr" Nov 01 23:07:55.371288 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:55.371201 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-77f6b5c75-k7qln" Nov 01 23:07:55.439687 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:55.439664 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-54876bc6b7-sj6gr" Nov 01 23:07:55.671418 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:07:55.671381 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e9a6cde_55d0_40d6_8fe7_e54452ef8da7.slice/crio-0af7f8932da8f3704a0634787cc81e46d994c287170ddbd20a02a5f55bc91152 WatchSource:0}: Error finding container 0af7f8932da8f3704a0634787cc81e46d994c287170ddbd20a02a5f55bc91152: Status 404 returned error can't find the container with id 0af7f8932da8f3704a0634787cc81e46d994c287170ddbd20a02a5f55bc91152 Nov 01 23:07:55.672018 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:55.671990 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-77f6b5c75-k7qln"] Nov 01 23:07:55.742580 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:55.742544 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-01e30d-54876bc6b7-sj6gr"] Nov 01 23:07:55.894877 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:55.893767 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2f3705-855984bb47-ztmpt" Nov 01 23:07:56.072503 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:56.072458 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583-kube-api-access-mw5lh\" (UniqueName: \"kubernetes.io/projected/c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583-kube-api-access-mw5lh\") pod \"c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583\" (UID: \"c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583\") " Nov 01 23:07:56.072753 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:56.072732 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583-plugins-config-volume\") pod \"c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583\" (UID: \"c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583\") " Nov 01 23:07:56.126860 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:56.115856 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583-plugins-config-volume" pod "c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583" (UID: "c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:07:56.144685 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:56.141158 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583-kube-api-access-mw5lh" pod "c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583" (UID: "c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583"). InnerVolumeSpecName "kube-api-access-mw5lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:56.180147 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:56.179938 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw5lh\" (UniqueName: \"kubernetes.io/projected/c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583-kube-api-access-mw5lh\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:56.180517 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:56.180492 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:56.183595 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:56.183563 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-77f6b5c75-k7qln" event={"ID":"5e9a6cde-55d0-40d6-8fe7-e54452ef8da7","Type":"ContainerStarted","Data":"f6ba54f67443c6ec7ff9f2914df9b68db2d4882045204b53030ebbf0379f8029"} Nov 01 23:07:56.183795 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:56.183775 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-77f6b5c75-k7qln" event={"ID":"5e9a6cde-55d0-40d6-8fe7-e54452ef8da7","Type":"ContainerStarted","Data":"0af7f8932da8f3704a0634787cc81e46d994c287170ddbd20a02a5f55bc91152"} Nov 01 23:07:56.185030 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:56.185010 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-77f6b5c75-k7qln" Nov 01 23:07:56.187471 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:56.187451 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-54876bc6b7-sj6gr" event={"ID":"a6207c82-7f15-4c9a-9012-5c1aa1659dd0","Type":"ContainerStarted","Data":"be637cd05086e2e00d013331fdfb77f9681cf7235833f9fdb70354bece8f600f"} Nov 01 23:07:56.187623 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:56.187610 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-54876bc6b7-sj6gr" event={"ID":"a6207c82-7f15-4c9a-9012-5c1aa1659dd0","Type":"ContainerStarted","Data":"cfb19df34ddd54ccd05b3f0936bce9e72c7af39c7d0fe827f1bf00c58a35d6bb"} Nov 01 23:07:56.188431 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:56.188413 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-54876bc6b7-sj6gr" Nov 01 23:07:56.193849 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:56.191975 5011 generic.go:356] "Generic (PLEG): container finished" podID="c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583" containerID="3e841d9918502a08128a3d7cbf9ee34e3cef12fb39a1bd5651109c7d1cc9d9c9" exitCode=0 Nov 01 23:07:56.193849 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:56.192120 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2f3705-855984bb47-ztmpt" event={"ID":"c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583","Type":"ContainerDied","Data":"3e841d9918502a08128a3d7cbf9ee34e3cef12fb39a1bd5651109c7d1cc9d9c9"} Nov 01 23:07:56.193849 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:56.192164 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2f3705-855984bb47-ztmpt" event={"ID":"c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583","Type":"ContainerDied","Data":"7573b4c739245a5dfdefeedfcd836ca5e69810890b0a08a76c110566c88b8585"} Nov 01 23:07:56.193849 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:56.192194 5011 scope.go:122] "RemoveContainer" containerID="3e841d9918502a08128a3d7cbf9ee34e3cef12fb39a1bd5651109c7d1cc9d9c9" Nov 01 23:07:56.193849 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:56.192339 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2f3705-855984bb47-ztmpt" Nov 01 23:07:56.225879 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:56.222150 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-77f6b5c75-k7qln" podStartSLOduration=2.222130419 podStartE2EDuration="2.222130419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:07:56.216198124 +0000 UTC m=+2644.620788757" watchObservedRunningTime="2026-11-01 23:07:56.222130419 +0000 UTC m=+2644.626721073" Nov 01 23:07:56.238033 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:56.237983 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-54876bc6b7-sj6gr" podStartSLOduration=2.237963868 podStartE2EDuration="2.237963868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:07:56.233963829 +0000 UTC m=+2644.638554465" watchObservedRunningTime="2026-11-01 23:07:56.237963868 +0000 UTC m=+2644.642554505" Nov 01 23:07:56.253160 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:56.253136 5011 scope.go:122] "RemoveContainer" containerID="3e841d9918502a08128a3d7cbf9ee34e3cef12fb39a1bd5651109c7d1cc9d9c9" Nov 01 23:07:56.255168 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:07:56.255139 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e841d9918502a08128a3d7cbf9ee34e3cef12fb39a1bd5651109c7d1cc9d9c9\": container with ID starting with 3e841d9918502a08128a3d7cbf9ee34e3cef12fb39a1bd5651109c7d1cc9d9c9 not found: ID does not exist" containerID="3e841d9918502a08128a3d7cbf9ee34e3cef12fb39a1bd5651109c7d1cc9d9c9" Nov 01 23:07:56.255371 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:56.255343 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e841d9918502a08128a3d7cbf9ee34e3cef12fb39a1bd5651109c7d1cc9d9c9"} err="failed to get container status \"3e841d9918502a08128a3d7cbf9ee34e3cef12fb39a1bd5651109c7d1cc9d9c9\": rpc error: code = NotFound desc = could not find container \"3e841d9918502a08128a3d7cbf9ee34e3cef12fb39a1bd5651109c7d1cc9d9c9\": container with ID starting with 3e841d9918502a08128a3d7cbf9ee34e3cef12fb39a1bd5651109c7d1cc9d9c9 not found: ID does not exist" Nov 01 23:07:56.255802 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:56.255782 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2f3705-855984bb47-ztmpt"] Nov 01 23:07:56.259998 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:56.259975 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2f3705-855984bb47-ztmpt"] Nov 01 23:07:57.749505 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:07:57.749473 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583" path="/var/lib/kubelet/pods/c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583/volumes" Nov 01 23:08:00.339615 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:00.339583 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5"] Nov 01 23:08:00.340143 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:00.339779 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5" podUID="91254bf5-2625-4e5b-bbf1-c802b51b8438" containerName="maas-api" containerID="cri-o://eec05dce6f13cb05a05683516241673e24a875e90182bf72bb9e71eeede71eb8" gracePeriod=30 Nov 01 23:08:00.904291 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:00.904248 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5" Nov 01 23:08:01.004038 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:01.004003 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91254bf5-2625-4e5b-bbf1-c802b51b8438-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91254bf5-2625-4e5b-bbf1-c802b51b8438-openshift-service-ca\") pod \"91254bf5-2625-4e5b-bbf1-c802b51b8438\" (UID: \"91254bf5-2625-4e5b-bbf1-c802b51b8438\") " Nov 01 23:08:01.004250 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:01.004089 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91254bf5-2625-4e5b-bbf1-c802b51b8438-metrics-certs\" (UniqueName: \"kubernetes.io/secret/91254bf5-2625-4e5b-bbf1-c802b51b8438-metrics-certs\") pod \"91254bf5-2625-4e5b-bbf1-c802b51b8438\" (UID: \"91254bf5-2625-4e5b-bbf1-c802b51b8438\") " Nov 01 23:08:01.004250 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:01.004176 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91254bf5-2625-4e5b-bbf1-c802b51b8438-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91254bf5-2625-4e5b-bbf1-c802b51b8438-maas-api-tls\") pod \"91254bf5-2625-4e5b-bbf1-c802b51b8438\" (UID: \"91254bf5-2625-4e5b-bbf1-c802b51b8438\") " Nov 01 23:08:01.004365 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:01.004304 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91254bf5-2625-4e5b-bbf1-c802b51b8438-kube-api-access-7gc94\" (UniqueName: \"kubernetes.io/projected/91254bf5-2625-4e5b-bbf1-c802b51b8438-kube-api-access-7gc94\") pod \"91254bf5-2625-4e5b-bbf1-c802b51b8438\" (UID: \"91254bf5-2625-4e5b-bbf1-c802b51b8438\") " Nov 01 23:08:01.004877 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:01.004826 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91254bf5-2625-4e5b-bbf1-c802b51b8438-openshift-service-ca" pod "91254bf5-2625-4e5b-bbf1-c802b51b8438" (UID: "91254bf5-2625-4e5b-bbf1-c802b51b8438"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:08:01.005051 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:01.005034 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91254bf5-2625-4e5b-bbf1-c802b51b8438-openshift-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:01.007350 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:01.007318 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91254bf5-2625-4e5b-bbf1-c802b51b8438-metrics-certs" pod "91254bf5-2625-4e5b-bbf1-c802b51b8438" (UID: "91254bf5-2625-4e5b-bbf1-c802b51b8438"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:08:01.008602 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:01.008581 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91254bf5-2625-4e5b-bbf1-c802b51b8438-maas-api-tls" pod "91254bf5-2625-4e5b-bbf1-c802b51b8438" (UID: "91254bf5-2625-4e5b-bbf1-c802b51b8438"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:08:01.008727 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:01.008712 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91254bf5-2625-4e5b-bbf1-c802b51b8438-kube-api-access-7gc94" pod "91254bf5-2625-4e5b-bbf1-c802b51b8438" (UID: "91254bf5-2625-4e5b-bbf1-c802b51b8438"). InnerVolumeSpecName "kube-api-access-7gc94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:08:01.107067 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:01.106987 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91254bf5-2625-4e5b-bbf1-c802b51b8438-maas-api-tls\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:01.107067 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:01.107016 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gc94\" (UniqueName: \"kubernetes.io/projected/91254bf5-2625-4e5b-bbf1-c802b51b8438-kube-api-access-7gc94\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:01.107067 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:01.107027 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91254bf5-2625-4e5b-bbf1-c802b51b8438-metrics-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:01.298300 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:01.298262 5011 generic.go:356] "Generic (PLEG): container finished" podID="91254bf5-2625-4e5b-bbf1-c802b51b8438" containerID="eec05dce6f13cb05a05683516241673e24a875e90182bf72bb9e71eeede71eb8" exitCode=0 Nov 01 23:08:01.298482 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:01.298325 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5" Nov 01 23:08:01.298482 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:01.298345 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5" event={"ID":"91254bf5-2625-4e5b-bbf1-c802b51b8438","Type":"ContainerDied","Data":"eec05dce6f13cb05a05683516241673e24a875e90182bf72bb9e71eeede71eb8"} Nov 01 23:08:01.298482 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:01.298409 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5" event={"ID":"91254bf5-2625-4e5b-bbf1-c802b51b8438","Type":"ContainerDied","Data":"807475c0de97b9f7fb3e7390d88b8f87f6d0105d0cb4c19f3e0830ed18de8aa8"} Nov 01 23:08:01.298482 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:01.298435 5011 scope.go:122] "RemoveContainer" containerID="eec05dce6f13cb05a05683516241673e24a875e90182bf72bb9e71eeede71eb8" Nov 01 23:08:01.310988 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:01.310965 5011 scope.go:122] "RemoveContainer" containerID="eec05dce6f13cb05a05683516241673e24a875e90182bf72bb9e71eeede71eb8" Nov 01 23:08:01.311333 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:08:01.311305 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eec05dce6f13cb05a05683516241673e24a875e90182bf72bb9e71eeede71eb8\": container with ID starting with eec05dce6f13cb05a05683516241673e24a875e90182bf72bb9e71eeede71eb8 not found: ID does not exist" containerID="eec05dce6f13cb05a05683516241673e24a875e90182bf72bb9e71eeede71eb8" Nov 01 23:08:01.311447 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:01.311344 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eec05dce6f13cb05a05683516241673e24a875e90182bf72bb9e71eeede71eb8"} err="failed to get container status \"eec05dce6f13cb05a05683516241673e24a875e90182bf72bb9e71eeede71eb8\": rpc error: code = NotFound desc = could not find container \"eec05dce6f13cb05a05683516241673e24a875e90182bf72bb9e71eeede71eb8\": container with ID starting with eec05dce6f13cb05a05683516241673e24a875e90182bf72bb9e71eeede71eb8 not found: ID does not exist" Nov 01 23:08:01.327045 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:01.327016 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5"] Nov 01 23:08:01.334195 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:01.334168 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-2f3705-868b8597fb-r2sg5"] Nov 01 23:08:01.786803 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:01.786771 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91254bf5-2625-4e5b-bbf1-c802b51b8438" path="/var/lib/kubelet/pods/91254bf5-2625-4e5b-bbf1-c802b51b8438/volumes" Nov 01 23:08:02.325455 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.325417 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5"] Nov 01 23:08:02.326658 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.326634 5011 cpu_manager.go:414] "containerMap: removing container" podUID="c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583" containerName="payload-pre-processing" Nov 01 23:08:02.326658 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.326659 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583" containerName="payload-pre-processing" Nov 01 23:08:02.326806 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.326705 5011 cpu_manager.go:414] "containerMap: removing container" podUID="91254bf5-2625-4e5b-bbf1-c802b51b8438" containerName="maas-api" Nov 01 23:08:02.326806 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.326712 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91254bf5-2625-4e5b-bbf1-c802b51b8438" containerName="maas-api" Nov 01 23:08:02.326930 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.326917 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5cfa8f2-966b-4b28-b8ff-0c23d4fd2583" containerName="payload-pre-processing" Nov 01 23:08:02.326972 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.326941 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="91254bf5-2625-4e5b-bbf1-c802b51b8438" containerName="maas-api" Nov 01 23:08:02.332587 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.332567 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:02.388881 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.386653 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5"] Nov 01 23:08:02.430967 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.430928 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istio-data\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:02.431149 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.431094 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-credential-socket\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:02.431149 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.431130 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istio-envoy\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:02.431229 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.431207 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istio-token\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:02.431275 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.431227 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istiod-ca-cert\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:02.431417 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.431391 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-952p6\" (UniqueName: \"kubernetes.io/projected/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-kube-api-access-952p6\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:02.431557 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.431532 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istio-podinfo\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:02.431726 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.431636 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-workload-socket\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:02.431850 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.431776 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-workload-certs\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:02.533663 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.533623 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istio-token\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:02.533663 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.533662 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istiod-ca-cert\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:02.533968 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.533691 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-952p6\" (UniqueName: \"kubernetes.io/projected/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-kube-api-access-952p6\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:02.533968 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.533762 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istio-podinfo\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:02.533968 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.533813 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-workload-socket\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:02.533968 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.533942 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-workload-certs\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:02.534233 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.534002 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istio-data\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:02.534233 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.534118 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-credential-socket\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:02.534233 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.534171 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istio-envoy\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:02.534408 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.534306 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-workload-socket\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:02.534408 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.534350 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-workload-certs\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:02.534476 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.534403 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istio-data\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:02.534476 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.534424 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istiod-ca-cert\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:02.534569 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.534533 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-credential-socket\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:02.536638 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.536617 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istio-podinfo\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:02.536766 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.536693 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istio-envoy\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:02.544064 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.544043 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istio-token\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:02.544356 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.544337 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-952p6\" (UniqueName: \"kubernetes.io/projected/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-kube-api-access-952p6\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:02.693859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:02.690622 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:03.023653 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:08:03.023623 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a2a9b4_ab92_4473_9afa_3a102b2babc4.slice/crio-457802f1d15b451bf9faf969c1845e308010575c84d32f89c715d8cb18acedfb WatchSource:0}: Error finding container 457802f1d15b451bf9faf969c1845e308010575c84d32f89c715d8cb18acedfb: Status 404 returned error can't find the container with id 457802f1d15b451bf9faf969c1845e308010575c84d32f89c715d8cb18acedfb Nov 01 23:08:03.024114 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:03.023975 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5"] Nov 01 23:08:03.026957 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:03.026923 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:08:03.027051 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:03.026994 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:08:03.027051 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:03.027025 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:08:03.324172 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:03.324074 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" event={"ID":"c7a2a9b4-ab92-4473-9afa-3a102b2babc4","Type":"ContainerStarted","Data":"0396a11d5b06e63d9ea5b20df3a253167c378df575f5cfb5b2dbbb574bf22d83"} Nov 01 23:08:03.324172 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:03.324148 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" event={"ID":"c7a2a9b4-ab92-4473-9afa-3a102b2babc4","Type":"ContainerStarted","Data":"457802f1d15b451bf9faf969c1845e308010575c84d32f89c715d8cb18acedfb"} Nov 01 23:08:03.354265 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:03.354210 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" podStartSLOduration=1.354196882 podStartE2EDuration="1.354196882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:08:03.351126406 +0000 UTC m=+2651.755717038" watchObservedRunningTime="2026-11-01 23:08:03.354196882 +0000 UTC m=+2651.758787517" Nov 01 23:08:03.691254 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:03.691224 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:04.703067 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:04.703015 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 01 23:08:04.703746 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:04.703692 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" podUID="c7a2a9b4-ab92-4473-9afa-3a102b2babc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 01 23:08:05.691878 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:05.691825 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 01 23:08:05.692083 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:05.691907 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" podUID="c7a2a9b4-ab92-4473-9afa-3a102b2babc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 01 23:08:06.704612 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:06.704577 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 01 23:08:06.705434 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:06.705401 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" podUID="c7a2a9b4-ab92-4473-9afa-3a102b2babc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 01 23:08:07.691290 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:07.691256 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 01 23:08:07.691503 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:07.691321 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" podUID="c7a2a9b4-ab92-4473-9afa-3a102b2babc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 01 23:08:08.270588 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:08.270552 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-77f6b5c75-k7qln" Nov 01 23:08:08.273030 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:08.273001 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-54876bc6b7-sj6gr" Nov 01 23:08:08.288870 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:08.288815 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j"] Nov 01 23:08:08.305149 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:08.305077 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" podUID="bd446c47-d4df-4f36-8eac-541975b1375f" containerName="istio-proxy" containerID="cri-o://1db7a4eb79dc4fa252d00ca3b7c8a6e51dec5d4052867f632e8cca4a6bb3dc97" gracePeriod=30 Nov 01 23:08:08.695860 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:08.693347 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Nov 01 23:08:08.695860 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:08.693414 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" podUID="c7a2a9b4-ab92-4473-9afa-3a102b2babc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Nov 01 23:08:09.735544 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:09.735513 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:09.736139 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:09.736118 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:09.737258 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:09.737234 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:08:12.333500 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:12.333463 5011 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:08:12.334010 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:12.333538 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" podUID="bd446c47-d4df-4f36-8eac-541975b1375f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:08:13.441916 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:13.441882 5011 generic.go:356] "Generic (PLEG): container finished" podID="bd446c47-d4df-4f36-8eac-541975b1375f" containerID="1db7a4eb79dc4fa252d00ca3b7c8a6e51dec5d4052867f632e8cca4a6bb3dc97" exitCode=0 Nov 01 23:08:13.442329 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:13.441950 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" event={"ID":"bd446c47-d4df-4f36-8eac-541975b1375f","Type":"ContainerDied","Data":"1db7a4eb79dc4fa252d00ca3b7c8a6e51dec5d4052867f632e8cca4a6bb3dc97"} Nov 01 23:08:13.664310 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:13.664283 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 23:08:13.827692 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:13.827657 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd446c47-d4df-4f36-8eac-541975b1375f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd446c47-d4df-4f36-8eac-541975b1375f-istio-podinfo\") pod \"bd446c47-d4df-4f36-8eac-541975b1375f\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " Nov 01 23:08:13.827908 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:13.827715 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-workload-certs\") pod \"bd446c47-d4df-4f36-8eac-541975b1375f\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " Nov 01 23:08:13.827908 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:13.827759 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-istio-envoy\") pod \"bd446c47-d4df-4f36-8eac-541975b1375f\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " Nov 01 23:08:13.827908 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:13.827784 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd446c47-d4df-4f36-8eac-541975b1375f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd446c47-d4df-4f36-8eac-541975b1375f-istiod-ca-cert\") pod \"bd446c47-d4df-4f36-8eac-541975b1375f\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " Nov 01 23:08:13.827908 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:13.827851 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-workload-socket\") pod \"bd446c47-d4df-4f36-8eac-541975b1375f\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " Nov 01 23:08:13.828157 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:13.827928 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd446c47-d4df-4f36-8eac-541975b1375f-kube-api-access-gjswv\" (UniqueName: \"kubernetes.io/projected/bd446c47-d4df-4f36-8eac-541975b1375f-kube-api-access-gjswv\") pod \"bd446c47-d4df-4f36-8eac-541975b1375f\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " Nov 01 23:08:13.828157 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:13.827964 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd446c47-d4df-4f36-8eac-541975b1375f-istio-token\" (UniqueName: \"kubernetes.io/projected/bd446c47-d4df-4f36-8eac-541975b1375f-istio-token\") pod \"bd446c47-d4df-4f36-8eac-541975b1375f\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " Nov 01 23:08:13.828157 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:13.828027 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-istio-data\") pod \"bd446c47-d4df-4f36-8eac-541975b1375f\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " Nov 01 23:08:13.828157 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:13.828072 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-credential-socket\") pod \"bd446c47-d4df-4f36-8eac-541975b1375f\" (UID: \"bd446c47-d4df-4f36-8eac-541975b1375f\") " Nov 01 23:08:13.828157 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:13.828113 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-workload-socket" pod "bd446c47-d4df-4f36-8eac-541975b1375f" (UID: "bd446c47-d4df-4f36-8eac-541975b1375f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:13.828544 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:13.828226 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd446c47-d4df-4f36-8eac-541975b1375f-istiod-ca-cert" pod "bd446c47-d4df-4f36-8eac-541975b1375f" (UID: "bd446c47-d4df-4f36-8eac-541975b1375f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:08:13.828544 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:13.828352 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-istio-data" pod "bd446c47-d4df-4f36-8eac-541975b1375f" (UID: "bd446c47-d4df-4f36-8eac-541975b1375f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:13.828544 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:13.828531 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-credential-socket" pod "bd446c47-d4df-4f36-8eac-541975b1375f" (UID: "bd446c47-d4df-4f36-8eac-541975b1375f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:13.828919 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:13.828895 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-workload-certs" pod "bd446c47-d4df-4f36-8eac-541975b1375f" (UID: "bd446c47-d4df-4f36-8eac-541975b1375f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:13.829443 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:13.829341 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:13.829443 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:13.829367 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd446c47-d4df-4f36-8eac-541975b1375f-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:13.829443 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:13.829382 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:13.829443 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:13.829397 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:13.829443 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:13.829412 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:13.830439 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:13.830416 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-istio-envoy" pod "bd446c47-d4df-4f36-8eac-541975b1375f" (UID: "bd446c47-d4df-4f36-8eac-541975b1375f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:13.830680 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:13.830653 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd446c47-d4df-4f36-8eac-541975b1375f-kube-api-access-gjswv" pod "bd446c47-d4df-4f36-8eac-541975b1375f" (UID: "bd446c47-d4df-4f36-8eac-541975b1375f"). InnerVolumeSpecName "kube-api-access-gjswv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:08:13.830898 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:13.830873 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd446c47-d4df-4f36-8eac-541975b1375f-istio-token" pod "bd446c47-d4df-4f36-8eac-541975b1375f" (UID: "bd446c47-d4df-4f36-8eac-541975b1375f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:08:13.831131 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:13.831113 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd446c47-d4df-4f36-8eac-541975b1375f-istio-podinfo" pod "bd446c47-d4df-4f36-8eac-541975b1375f" (UID: "bd446c47-d4df-4f36-8eac-541975b1375f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:08:13.930700 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:13.930666 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd446c47-d4df-4f36-8eac-541975b1375f-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:13.930700 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:13.930698 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd446c47-d4df-4f36-8eac-541975b1375f-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:13.930927 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:13.930707 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjswv\" (UniqueName: \"kubernetes.io/projected/bd446c47-d4df-4f36-8eac-541975b1375f-kube-api-access-gjswv\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:13.930927 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:13.930717 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd446c47-d4df-4f36-8eac-541975b1375f-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:14.451590 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:14.451545 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" event={"ID":"bd446c47-d4df-4f36-8eac-541975b1375f","Type":"ContainerDied","Data":"62e39f53801050cbef9d78e84e492e9aaf5dbebacaa97439f2e8ffcdec052c0a"} Nov 01 23:08:14.452074 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:14.451634 5011 scope.go:122] "RemoveContainer" containerID="1db7a4eb79dc4fa252d00ca3b7c8a6e51dec5d4052867f632e8cca4a6bb3dc97" Nov 01 23:08:14.452074 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:14.451566 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j" Nov 01 23:08:14.482386 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:14.482333 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j"] Nov 01 23:08:14.486171 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:14.486143 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-k6c4j"] Nov 01 23:08:15.755060 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:15.755025 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd446c47-d4df-4f36-8eac-541975b1375f" path="/var/lib/kubelet/pods/bd446c47-d4df-4f36-8eac-541975b1375f/volumes" Nov 01 23:08:16.962279 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:08:16.962228 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:08:29.209638 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:29.209599 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-49ab43-696d67bfd7-mkrrg"] Nov 01 23:08:29.226994 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:29.226958 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-49ab43-5bc95ddff9-m8bzq"] Nov 01 23:08:34.210326 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:34.210281 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-49ab43-696d67bfd7-mkrrg" podUID="94e3321d-ea40-40e4-9c13-d55c83c02e61" containerName="payload-processing" containerID="cri-o://1e851d4e1015415514f5223640bb8bc9be71f80b21450401c4f147409cf92d10" gracePeriod=25 Nov 01 23:08:34.227551 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:34.227494 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-49ab43-5bc95ddff9-m8bzq" podUID="c828f45b-c1cd-4f21-8ab1-b050cba1a54e" containerName="payload-pre-processing" containerID="cri-o://a72ec049be112cb0903f4f3108f301af12db7222bf7461520215faafd447c212" gracePeriod=25 Nov 01 23:08:35.294632 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.294592 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm"] Nov 01 23:08:35.295936 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.295913 5011 cpu_manager.go:414] "containerMap: removing container" podUID="bd446c47-d4df-4f36-8eac-541975b1375f" containerName="istio-proxy" Nov 01 23:08:35.296135 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.296120 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd446c47-d4df-4f36-8eac-541975b1375f" containerName="istio-proxy" Nov 01 23:08:35.296517 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.296498 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd446c47-d4df-4f36-8eac-541975b1375f" containerName="istio-proxy" Nov 01 23:08:35.383955 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.383924 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm"] Nov 01 23:08:35.384143 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.384049 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:35.507069 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.507022 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a32d183e-1080-4cef-ab97-f31aa667ef5c-istio-token\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:35.507258 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.507165 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a32d183e-1080-4cef-ab97-f31aa667ef5c-istiod-ca-cert\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:35.507258 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.507250 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-workload-socket\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:35.507410 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.507274 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94bsm\" (UniqueName: \"kubernetes.io/projected/a32d183e-1080-4cef-ab97-f31aa667ef5c-kube-api-access-94bsm\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:35.507483 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.507434 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-workload-certs\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:35.507483 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.507472 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-credential-socket\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:35.507698 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.507674 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-istio-envoy\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:35.507740 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.507717 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-istio-data\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:35.507897 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.507880 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a32d183e-1080-4cef-ab97-f31aa667ef5c-istio-podinfo\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:35.609341 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.609255 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a32d183e-1080-4cef-ab97-f31aa667ef5c-istiod-ca-cert\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:35.609341 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.609310 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-workload-socket\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:35.609341 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.609333 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94bsm\" (UniqueName: \"kubernetes.io/projected/a32d183e-1080-4cef-ab97-f31aa667ef5c-kube-api-access-94bsm\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:35.609627 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.609399 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-workload-certs\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:35.609627 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.609426 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-credential-socket\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:35.609627 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.609564 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-istio-envoy\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:35.609627 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.609596 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-istio-data\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:35.609879 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.609707 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a32d183e-1080-4cef-ab97-f31aa667ef5c-istio-podinfo\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:35.609879 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.609732 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-workload-socket\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:35.609879 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.609787 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-workload-certs\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:35.609879 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.609856 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a32d183e-1080-4cef-ab97-f31aa667ef5c-istio-token\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:35.609879 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.609867 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-credential-socket\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:35.610138 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.610014 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-istio-data\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:35.610348 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.610305 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a32d183e-1080-4cef-ab97-f31aa667ef5c-istiod-ca-cert\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:35.612106 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.612079 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-istio-envoy\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:35.612308 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.612291 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a32d183e-1080-4cef-ab97-f31aa667ef5c-istio-podinfo\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:35.625936 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.625909 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94bsm\" (UniqueName: \"kubernetes.io/projected/a32d183e-1080-4cef-ab97-f31aa667ef5c-kube-api-access-94bsm\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:35.626103 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.626072 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a32d183e-1080-4cef-ab97-f31aa667ef5c-istio-token\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:35.796471 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:35.796438 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:36.264487 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:08:36.264455 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda32d183e_1080_4cef_ab97_f31aa667ef5c.slice/crio-0985bd22a9f314a1f9b1edd7c743e39a6174935247f9c4a32853c7f6124fbf9a WatchSource:0}: Error finding container 0985bd22a9f314a1f9b1edd7c743e39a6174935247f9c4a32853c7f6124fbf9a: Status 404 returned error can't find the container with id 0985bd22a9f314a1f9b1edd7c743e39a6174935247f9c4a32853c7f6124fbf9a Nov 01 23:08:36.264634 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:36.264488 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm"] Nov 01 23:08:36.267705 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:36.267670 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:08:36.267799 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:36.267738 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:08:36.267799 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:36.267770 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:08:36.728251 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:36.728216 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" event={"ID":"a32d183e-1080-4cef-ab97-f31aa667ef5c","Type":"ContainerStarted","Data":"b7f936c0b489a526ca56a2758ae47e11f71ff3fbbfc7538a43523b13be5b198b"} Nov 01 23:08:36.728678 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:36.728289 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" event={"ID":"a32d183e-1080-4cef-ab97-f31aa667ef5c","Type":"ContainerStarted","Data":"0985bd22a9f314a1f9b1edd7c743e39a6174935247f9c4a32853c7f6124fbf9a"} Nov 01 23:08:36.756598 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:36.756541 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" podStartSLOduration=1.7565263949999999 podStartE2EDuration="1.756526395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:08:36.750302113 +0000 UTC m=+2685.154892748" watchObservedRunningTime="2026-11-01 23:08:36.756526395 +0000 UTC m=+2685.161117026" Nov 01 23:08:36.797649 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:36.797611 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:36.954358 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:36.954023 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-49ab43-696d67bfd7-mkrrg" podUID="94e3321d-ea40-40e4-9c13-d55c83c02e61" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:08:36.954685 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:36.954368 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-49ab43-5bc95ddff9-m8bzq" podUID="c828f45b-c1cd-4f21-8ab1-b050cba1a54e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:08:37.798316 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:37.798272 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Nov 01 23:08:37.798793 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:37.798360 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" podUID="a32d183e-1080-4cef-ab97-f31aa667ef5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Nov 01 23:08:38.386664 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:38.385371 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": read tcp 10.133.0.2:48526->10.133.0.184:15021: read: connection reset by peer" start-of-body= Nov 01 23:08:38.386664 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:38.385439 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" podUID="bc7cb565-6bf1-400f-9a3d-f062af8270bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": read tcp 10.133.0.2:48526->10.133.0.184:15021: read: connection reset by peer" Nov 01 23:08:38.786884 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:38.785891 5011 generic.go:356] "Generic (PLEG): container finished" podID="bc7cb565-6bf1-400f-9a3d-f062af8270bd" containerID="ffbe0a2f2c8528da3bf0856546f4e257f4225f6f4875da3182c647330c97d2dc" exitCode=0 Nov 01 23:08:38.787097 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:38.786979 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" event={"ID":"bc7cb565-6bf1-400f-9a3d-f062af8270bd","Type":"ContainerDied","Data":"ffbe0a2f2c8528da3bf0856546f4e257f4225f6f4875da3182c647330c97d2dc"} Nov 01 23:08:38.787513 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:38.787488 5011 scope.go:122] "RemoveContainer" containerID="ffbe0a2f2c8528da3bf0856546f4e257f4225f6f4875da3182c647330c97d2dc" Nov 01 23:08:38.790062 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:38.790026 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:08:38.790162 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:38.790117 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:08:38.790205 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:38.790169 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:08:38.797481 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:38.797456 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Nov 01 23:08:38.797578 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:38.797504 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" podUID="a32d183e-1080-4cef-ab97-f31aa667ef5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Nov 01 23:08:39.091540 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:39.089646 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-01e30d-54876bc6b7-sj6gr"] Nov 01 23:08:39.091540 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:39.091186 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-77f6b5c75-k7qln"] Nov 01 23:08:39.798868 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:39.797490 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Nov 01 23:08:39.798868 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:39.797545 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" podUID="a32d183e-1080-4cef-ab97-f31aa667ef5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Nov 01 23:08:39.805904 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:39.805210 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" event={"ID":"bc7cb565-6bf1-400f-9a3d-f062af8270bd","Type":"ContainerStarted","Data":"6df3bb6dd4358e48f785522540e63f02db67a98bdd1a32f63b07b721946ef51c"} Nov 01 23:08:40.697858 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:40.697464 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:08:40.698392 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:40.697928 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 01 23:08:40.698392 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:40.697999 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" podUID="bc7cb565-6bf1-400f-9a3d-f062af8270bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 01 23:08:40.797386 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:40.797040 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Nov 01 23:08:40.797386 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:40.797091 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" podUID="a32d183e-1080-4cef-ab97-f31aa667ef5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Nov 01 23:08:41.697713 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:41.697653 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 01 23:08:41.698035 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:41.697721 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" podUID="bc7cb565-6bf1-400f-9a3d-f062af8270bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 01 23:08:41.813986 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:41.813949 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Nov 01 23:08:41.828917 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:41.814245 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" podUID="a32d183e-1080-4cef-ab97-f31aa667ef5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Nov 01 23:08:42.699308 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:42.699275 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 01 23:08:42.701766 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:42.701352 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" podUID="bc7cb565-6bf1-400f-9a3d-f062af8270bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 01 23:08:42.797340 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:42.797306 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Nov 01 23:08:42.797607 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:42.797581 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" podUID="a32d183e-1080-4cef-ab97-f31aa667ef5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Nov 01 23:08:43.698755 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:43.698703 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 01 23:08:43.699118 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:43.699072 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" podUID="bc7cb565-6bf1-400f-9a3d-f062af8270bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 01 23:08:43.797394 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:43.797361 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Nov 01 23:08:43.797852 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:43.797428 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" podUID="a32d183e-1080-4cef-ab97-f31aa667ef5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Nov 01 23:08:44.090606 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.090522 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-54876bc6b7-sj6gr" podUID="a6207c82-7f15-4c9a-9012-5c1aa1659dd0" containerName="payload-processing" containerID="cri-o://be637cd05086e2e00d013331fdfb77f9681cf7235833f9fdb70354bece8f600f" gracePeriod=25 Nov 01 23:08:44.091988 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.091954 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-77f6b5c75-k7qln" podUID="5e9a6cde-55d0-40d6-8fe7-e54452ef8da7" containerName="payload-pre-processing" containerID="cri-o://f6ba54f67443c6ec7ff9f2914df9b68db2d4882045204b53030ebbf0379f8029" gracePeriod=25 Nov 01 23:08:44.540190 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.540160 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-54876bc6b7-sj6gr" Nov 01 23:08:44.620854 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.620793 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-77f6b5c75-k7qln" Nov 01 23:08:44.679990 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.679931 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e9a6cde-55d0-40d6-8fe7-e54452ef8da7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e9a6cde-55d0-40d6-8fe7-e54452ef8da7-plugins-config-volume\") pod \"5e9a6cde-55d0-40d6-8fe7-e54452ef8da7\" (UID: \"5e9a6cde-55d0-40d6-8fe7-e54452ef8da7\") " Nov 01 23:08:44.680244 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.680223 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6207c82-7f15-4c9a-9012-5c1aa1659dd0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6207c82-7f15-4c9a-9012-5c1aa1659dd0-plugins-config-volume\") pod \"a6207c82-7f15-4c9a-9012-5c1aa1659dd0\" (UID: \"a6207c82-7f15-4c9a-9012-5c1aa1659dd0\") " Nov 01 23:08:44.680340 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.680321 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e9a6cde-55d0-40d6-8fe7-e54452ef8da7-kube-api-access-dmzv7\" (UniqueName: \"kubernetes.io/projected/5e9a6cde-55d0-40d6-8fe7-e54452ef8da7-kube-api-access-dmzv7\") pod \"5e9a6cde-55d0-40d6-8fe7-e54452ef8da7\" (UID: \"5e9a6cde-55d0-40d6-8fe7-e54452ef8da7\") " Nov 01 23:08:44.680473 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.680452 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6207c82-7f15-4c9a-9012-5c1aa1659dd0-kube-api-access-lh9g6\" (UniqueName: \"kubernetes.io/projected/a6207c82-7f15-4c9a-9012-5c1aa1659dd0-kube-api-access-lh9g6\") pod \"a6207c82-7f15-4c9a-9012-5c1aa1659dd0\" (UID: \"a6207c82-7f15-4c9a-9012-5c1aa1659dd0\") " Nov 01 23:08:44.680643 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.680621 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e9a6cde-55d0-40d6-8fe7-e54452ef8da7-plugins-config-volume" pod "5e9a6cde-55d0-40d6-8fe7-e54452ef8da7" (UID: "5e9a6cde-55d0-40d6-8fe7-e54452ef8da7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:08:44.681086 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.681059 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6207c82-7f15-4c9a-9012-5c1aa1659dd0-plugins-config-volume" pod "a6207c82-7f15-4c9a-9012-5c1aa1659dd0" (UID: "a6207c82-7f15-4c9a-9012-5c1aa1659dd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:08:44.681295 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.681271 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e9a6cde-55d0-40d6-8fe7-e54452ef8da7-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:44.681374 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.681305 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6207c82-7f15-4c9a-9012-5c1aa1659dd0-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:44.683597 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.683573 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6207c82-7f15-4c9a-9012-5c1aa1659dd0-kube-api-access-lh9g6" pod "a6207c82-7f15-4c9a-9012-5c1aa1659dd0" (UID: "a6207c82-7f15-4c9a-9012-5c1aa1659dd0"). InnerVolumeSpecName "kube-api-access-lh9g6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:08:44.683709 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.683675 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e9a6cde-55d0-40d6-8fe7-e54452ef8da7-kube-api-access-dmzv7" pod "5e9a6cde-55d0-40d6-8fe7-e54452ef8da7" (UID: "5e9a6cde-55d0-40d6-8fe7-e54452ef8da7"). InnerVolumeSpecName "kube-api-access-dmzv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:08:44.697380 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.697348 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 01 23:08:44.697525 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.697410 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" podUID="bc7cb565-6bf1-400f-9a3d-f062af8270bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 01 23:08:44.783418 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.783336 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh9g6\" (UniqueName: \"kubernetes.io/projected/a6207c82-7f15-4c9a-9012-5c1aa1659dd0-kube-api-access-lh9g6\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:44.783418 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.783366 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmzv7\" (UniqueName: \"kubernetes.io/projected/5e9a6cde-55d0-40d6-8fe7-e54452ef8da7-kube-api-access-dmzv7\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:44.802700 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.802669 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:44.803127 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.803078 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:44.803943 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.803922 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:08:44.926429 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.926392 5011 generic.go:356] "Generic (PLEG): container finished" podID="5e9a6cde-55d0-40d6-8fe7-e54452ef8da7" containerID="f6ba54f67443c6ec7ff9f2914df9b68db2d4882045204b53030ebbf0379f8029" exitCode=0 Nov 01 23:08:44.926621 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.926448 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-77f6b5c75-k7qln" Nov 01 23:08:44.926621 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.926484 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-77f6b5c75-k7qln" event={"ID":"5e9a6cde-55d0-40d6-8fe7-e54452ef8da7","Type":"ContainerDied","Data":"f6ba54f67443c6ec7ff9f2914df9b68db2d4882045204b53030ebbf0379f8029"} Nov 01 23:08:44.926971 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.926939 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-77f6b5c75-k7qln" event={"ID":"5e9a6cde-55d0-40d6-8fe7-e54452ef8da7","Type":"ContainerDied","Data":"0af7f8932da8f3704a0634787cc81e46d994c287170ddbd20a02a5f55bc91152"} Nov 01 23:08:44.927102 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.926994 5011 scope.go:122] "RemoveContainer" containerID="f6ba54f67443c6ec7ff9f2914df9b68db2d4882045204b53030ebbf0379f8029" Nov 01 23:08:44.929334 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.929310 5011 generic.go:356] "Generic (PLEG): container finished" podID="a6207c82-7f15-4c9a-9012-5c1aa1659dd0" containerID="be637cd05086e2e00d013331fdfb77f9681cf7235833f9fdb70354bece8f600f" exitCode=0 Nov 01 23:08:44.929466 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.929349 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-54876bc6b7-sj6gr" Nov 01 23:08:44.929466 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.929401 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-54876bc6b7-sj6gr" event={"ID":"a6207c82-7f15-4c9a-9012-5c1aa1659dd0","Type":"ContainerDied","Data":"be637cd05086e2e00d013331fdfb77f9681cf7235833f9fdb70354bece8f600f"} Nov 01 23:08:44.929466 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.929454 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-54876bc6b7-sj6gr" event={"ID":"a6207c82-7f15-4c9a-9012-5c1aa1659dd0","Type":"ContainerDied","Data":"cfb19df34ddd54ccd05b3f0936bce9e72c7af39c7d0fe827f1bf00c58a35d6bb"} Nov 01 23:08:44.965863 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.964006 5011 scope.go:122] "RemoveContainer" containerID="f6ba54f67443c6ec7ff9f2914df9b68db2d4882045204b53030ebbf0379f8029" Nov 01 23:08:44.974167 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:08:44.970032 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6ba54f67443c6ec7ff9f2914df9b68db2d4882045204b53030ebbf0379f8029\": container with ID starting with f6ba54f67443c6ec7ff9f2914df9b68db2d4882045204b53030ebbf0379f8029 not found: ID does not exist" containerID="f6ba54f67443c6ec7ff9f2914df9b68db2d4882045204b53030ebbf0379f8029" Nov 01 23:08:44.974167 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.970095 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6ba54f67443c6ec7ff9f2914df9b68db2d4882045204b53030ebbf0379f8029"} err="failed to get container status \"f6ba54f67443c6ec7ff9f2914df9b68db2d4882045204b53030ebbf0379f8029\": rpc error: code = NotFound desc = could not find container \"f6ba54f67443c6ec7ff9f2914df9b68db2d4882045204b53030ebbf0379f8029\": container with ID starting with f6ba54f67443c6ec7ff9f2914df9b68db2d4882045204b53030ebbf0379f8029 not found: ID does not exist" Nov 01 23:08:44.974167 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.970124 5011 scope.go:122] "RemoveContainer" containerID="be637cd05086e2e00d013331fdfb77f9681cf7235833f9fdb70354bece8f600f" Nov 01 23:08:44.992057 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.992027 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-01e30d-54876bc6b7-sj6gr"] Nov 01 23:08:44.995370 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.995350 5011 scope.go:122] "RemoveContainer" containerID="be637cd05086e2e00d013331fdfb77f9681cf7235833f9fdb70354bece8f600f" Nov 01 23:08:44.995738 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:08:44.995712 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be637cd05086e2e00d013331fdfb77f9681cf7235833f9fdb70354bece8f600f\": container with ID starting with be637cd05086e2e00d013331fdfb77f9681cf7235833f9fdb70354bece8f600f not found: ID does not exist" containerID="be637cd05086e2e00d013331fdfb77f9681cf7235833f9fdb70354bece8f600f" Nov 01 23:08:44.995935 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.995902 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be637cd05086e2e00d013331fdfb77f9681cf7235833f9fdb70354bece8f600f"} err="failed to get container status \"be637cd05086e2e00d013331fdfb77f9681cf7235833f9fdb70354bece8f600f\": rpc error: code = NotFound desc = could not find container \"be637cd05086e2e00d013331fdfb77f9681cf7235833f9fdb70354bece8f600f\": container with ID starting with be637cd05086e2e00d013331fdfb77f9681cf7235833f9fdb70354bece8f600f not found: ID does not exist" Nov 01 23:08:44.997345 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:44.997322 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-01e30d-54876bc6b7-sj6gr"] Nov 01 23:08:45.007930 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:45.007906 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-77f6b5c75-k7qln"] Nov 01 23:08:45.012689 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:45.012667 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-77f6b5c75-k7qln"] Nov 01 23:08:45.697401 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:45.697364 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Nov 01 23:08:45.697574 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:45.697425 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" podUID="bc7cb565-6bf1-400f-9a3d-f062af8270bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Nov 01 23:08:45.774596 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:45.774560 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e9a6cde-55d0-40d6-8fe7-e54452ef8da7" path="/var/lib/kubelet/pods/5e9a6cde-55d0-40d6-8fe7-e54452ef8da7/volumes" Nov 01 23:08:45.775080 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:45.775066 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6207c82-7f15-4c9a-9012-5c1aa1659dd0" path="/var/lib/kubelet/pods/a6207c82-7f15-4c9a-9012-5c1aa1659dd0/volumes" Nov 01 23:08:46.701902 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:46.701872 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:08:46.702406 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:46.702384 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:08:46.703077 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:46.703054 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:08:46.924387 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:46.924350 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-49ab43-5bc95ddff9-m8bzq" podUID="c828f45b-c1cd-4f21-8ab1-b050cba1a54e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:08:46.924584 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:46.924355 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-49ab43-696d67bfd7-mkrrg" podUID="94e3321d-ea40-40e4-9c13-d55c83c02e61" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:08:46.962398 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:08:46.962318 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:08:47.736876 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:47.736783 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b"] Nov 01 23:08:47.737367 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:47.737097 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" podUID="e06dee51-d25d-4e3c-bd5b-30ceb2bac53c" containerName="istio-proxy" containerID="cri-o://4beabb43d847c1aa1aa3b68e330073fe0bca140ab287394e174f06a4b99bc58b" gracePeriod=30 Nov 01 23:08:48.835405 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:48.835370 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8"] Nov 01 23:08:48.835817 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:48.835552 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8" podUID="c56973f4-5487-4aa0-88e9-c0e0161fb912" containerName="maas-api" containerID="cri-o://3412f6d4012884f829c0c8d6bfe2256c6ec4a743943d3ed56fa1edb57105bc55" gracePeriod=30 Nov 01 23:08:48.996799 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:48.996754 5011 generic.go:356] "Generic (PLEG): container finished" podID="c56973f4-5487-4aa0-88e9-c0e0161fb912" containerID="3412f6d4012884f829c0c8d6bfe2256c6ec4a743943d3ed56fa1edb57105bc55" exitCode=0 Nov 01 23:08:48.997132 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:48.997110 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8" event={"ID":"c56973f4-5487-4aa0-88e9-c0e0161fb912","Type":"ContainerDied","Data":"3412f6d4012884f829c0c8d6bfe2256c6ec4a743943d3ed56fa1edb57105bc55"} Nov 01 23:08:49.357654 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:49.357616 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8" Nov 01 23:08:49.385600 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:49.385565 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c56973f4-5487-4aa0-88e9-c0e0161fb912-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56973f4-5487-4aa0-88e9-c0e0161fb912-openshift-service-ca\") pod \"c56973f4-5487-4aa0-88e9-c0e0161fb912\" (UID: \"c56973f4-5487-4aa0-88e9-c0e0161fb912\") " Nov 01 23:08:49.385897 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:49.385727 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c56973f4-5487-4aa0-88e9-c0e0161fb912-kube-api-access-tbmtx\" (UniqueName: \"kubernetes.io/projected/c56973f4-5487-4aa0-88e9-c0e0161fb912-kube-api-access-tbmtx\") pod \"c56973f4-5487-4aa0-88e9-c0e0161fb912\" (UID: \"c56973f4-5487-4aa0-88e9-c0e0161fb912\") " Nov 01 23:08:49.385897 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:49.385767 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c56973f4-5487-4aa0-88e9-c0e0161fb912-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c56973f4-5487-4aa0-88e9-c0e0161fb912-metrics-certs\") pod \"c56973f4-5487-4aa0-88e9-c0e0161fb912\" (UID: \"c56973f4-5487-4aa0-88e9-c0e0161fb912\") " Nov 01 23:08:49.385897 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:49.385882 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c56973f4-5487-4aa0-88e9-c0e0161fb912-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56973f4-5487-4aa0-88e9-c0e0161fb912-maas-api-tls\") pod \"c56973f4-5487-4aa0-88e9-c0e0161fb912\" (UID: \"c56973f4-5487-4aa0-88e9-c0e0161fb912\") " Nov 01 23:08:49.386098 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:49.386005 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c56973f4-5487-4aa0-88e9-c0e0161fb912-openshift-service-ca" pod "c56973f4-5487-4aa0-88e9-c0e0161fb912" (UID: "c56973f4-5487-4aa0-88e9-c0e0161fb912"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:08:49.386515 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:49.386489 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56973f4-5487-4aa0-88e9-c0e0161fb912-openshift-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:49.389000 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:49.388970 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c56973f4-5487-4aa0-88e9-c0e0161fb912-metrics-certs" pod "c56973f4-5487-4aa0-88e9-c0e0161fb912" (UID: "c56973f4-5487-4aa0-88e9-c0e0161fb912"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:08:49.389347 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:49.389302 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56973f4-5487-4aa0-88e9-c0e0161fb912-kube-api-access-tbmtx" pod "c56973f4-5487-4aa0-88e9-c0e0161fb912" (UID: "c56973f4-5487-4aa0-88e9-c0e0161fb912"). InnerVolumeSpecName "kube-api-access-tbmtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:08:49.389888 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:49.389857 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c56973f4-5487-4aa0-88e9-c0e0161fb912-maas-api-tls" pod "c56973f4-5487-4aa0-88e9-c0e0161fb912" (UID: "c56973f4-5487-4aa0-88e9-c0e0161fb912"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:08:49.487615 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:49.487575 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbmtx\" (UniqueName: \"kubernetes.io/projected/c56973f4-5487-4aa0-88e9-c0e0161fb912-kube-api-access-tbmtx\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:49.487615 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:49.487610 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c56973f4-5487-4aa0-88e9-c0e0161fb912-metrics-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:49.487615 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:49.487621 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56973f4-5487-4aa0-88e9-c0e0161fb912-maas-api-tls\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:50.013167 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:50.013052 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8" event={"ID":"c56973f4-5487-4aa0-88e9-c0e0161fb912","Type":"ContainerDied","Data":"94dfd39a0c678fa7e12785cbefa07924b2adb534e34a798af84d6d91f5fffc35"} Nov 01 23:08:50.013167 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:50.013123 5011 scope.go:122] "RemoveContainer" containerID="3412f6d4012884f829c0c8d6bfe2256c6ec4a743943d3ed56fa1edb57105bc55" Nov 01 23:08:50.013598 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:50.013289 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8" Nov 01 23:08:50.042572 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:50.042535 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8"] Nov 01 23:08:50.049640 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:50.049602 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8"] Nov 01 23:08:51.749385 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:51.749350 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c56973f4-5487-4aa0-88e9-c0e0161fb912" path="/var/lib/kubelet/pods/c56973f4-5487-4aa0-88e9-c0e0161fb912/volumes" Nov 01 23:08:52.299761 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:52.299713 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-49ab43-5f998dc7f9-vh5h8" podUID="c56973f4-5487-4aa0-88e9-c0e0161fb912" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.158:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 23:08:52.810746 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:52.810712 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Nov 01 23:08:52.811203 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:52.810767 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" podUID="e06dee51-d25d-4e3c-bd5b-30ceb2bac53c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Nov 01 23:08:53.062036 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:53.062007 5011 generic.go:356] "Generic (PLEG): container finished" podID="e06dee51-d25d-4e3c-bd5b-30ceb2bac53c" containerID="4beabb43d847c1aa1aa3b68e330073fe0bca140ab287394e174f06a4b99bc58b" exitCode=0 Nov 01 23:08:53.062171 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:53.062062 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" event={"ID":"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c","Type":"ContainerDied","Data":"4beabb43d847c1aa1aa3b68e330073fe0bca140ab287394e174f06a4b99bc58b"} Nov 01 23:08:53.062171 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:53.062134 5011 scope.go:122] "RemoveContainer" containerID="0a129b111f383ff4e3ff388091569fde4e1226c06fad58b5faf94d8e39b9faa7" Nov 01 23:08:53.088108 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:53.088082 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:08:53.164961 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:53.164928 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-workload-certs\") pod \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " Nov 01 23:08:53.165166 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:53.165053 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istio-token\" (UniqueName: \"kubernetes.io/projected/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istio-token\") pod \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " Nov 01 23:08:53.165166 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:53.165091 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istio-data\") pod \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " Nov 01 23:08:53.165166 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:53.165136 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-workload-certs" pod "e06dee51-d25d-4e3c-bd5b-30ceb2bac53c" (UID: "e06dee51-d25d-4e3c-bd5b-30ceb2bac53c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:53.165450 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:53.165432 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istio-data" pod "e06dee51-d25d-4e3c-bd5b-30ceb2bac53c" (UID: "e06dee51-d25d-4e3c-bd5b-30ceb2bac53c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:53.165556 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:53.165474 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istio-envoy\") pod \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " Nov 01 23:08:53.165688 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:53.165671 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-credential-socket\") pod \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " Nov 01 23:08:53.165767 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:53.165735 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-kube-api-access-9mzrm\" (UniqueName: \"kubernetes.io/projected/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-kube-api-access-9mzrm\") pod \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " Nov 01 23:08:53.165871 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:53.165783 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istio-podinfo\") pod \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " Nov 01 23:08:53.165871 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:53.165855 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istiod-ca-cert\") pod \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " Nov 01 23:08:53.165989 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:53.165915 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-workload-socket\") pod \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\" (UID: \"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c\") " Nov 01 23:08:53.166367 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:53.166220 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-credential-socket" pod "e06dee51-d25d-4e3c-bd5b-30ceb2bac53c" (UID: "e06dee51-d25d-4e3c-bd5b-30ceb2bac53c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:53.166464 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:53.166401 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-workload-socket" pod "e06dee51-d25d-4e3c-bd5b-30ceb2bac53c" (UID: "e06dee51-d25d-4e3c-bd5b-30ceb2bac53c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:53.166523 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:53.166475 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istiod-ca-cert" pod "e06dee51-d25d-4e3c-bd5b-30ceb2bac53c" (UID: "e06dee51-d25d-4e3c-bd5b-30ceb2bac53c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:08:53.166914 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:53.166888 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:53.166999 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:53.166918 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:53.166999 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:53.166935 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:53.166999 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:53.166947 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:53.166999 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:53.166960 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:53.168054 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:53.168026 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istio-token" pod "e06dee51-d25d-4e3c-bd5b-30ceb2bac53c" (UID: "e06dee51-d25d-4e3c-bd5b-30ceb2bac53c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:08:53.169539 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:53.169515 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-kube-api-access-9mzrm" pod "e06dee51-d25d-4e3c-bd5b-30ceb2bac53c" (UID: "e06dee51-d25d-4e3c-bd5b-30ceb2bac53c"). InnerVolumeSpecName "kube-api-access-9mzrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:08:53.172930 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:53.172906 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istio-envoy" pod "e06dee51-d25d-4e3c-bd5b-30ceb2bac53c" (UID: "e06dee51-d25d-4e3c-bd5b-30ceb2bac53c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:53.173018 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:53.172938 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istio-podinfo" pod "e06dee51-d25d-4e3c-bd5b-30ceb2bac53c" (UID: "e06dee51-d25d-4e3c-bd5b-30ceb2bac53c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:08:53.268690 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:53.268654 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mzrm\" (UniqueName: \"kubernetes.io/projected/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-kube-api-access-9mzrm\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:53.268690 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:53.268684 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:53.268690 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:53.268695 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:53.268981 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:53.268706 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:54.082785 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:54.082342 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" event={"ID":"e06dee51-d25d-4e3c-bd5b-30ceb2bac53c","Type":"ContainerDied","Data":"d7780b4ce41b363860e72af8f71ed10692ffe64f61bddacc180faac9a6d8b19a"} Nov 01 23:08:54.082785 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:54.082407 5011 scope.go:122] "RemoveContainer" containerID="4beabb43d847c1aa1aa3b68e330073fe0bca140ab287394e174f06a4b99bc58b" Nov 01 23:08:54.082785 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:54.082536 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b" Nov 01 23:08:54.149596 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:54.149562 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b"] Nov 01 23:08:54.153612 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:54.153588 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c00bbf-openshift-default-c86c5f9d9-frl4b"] Nov 01 23:08:54.604052 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:54.604028 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-49ab43-696d67bfd7-mkrrg" Nov 01 23:08:54.636411 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:54.636379 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-49ab43-5bc95ddff9-m8bzq" Nov 01 23:08:54.723037 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:54.722999 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94e3321d-ea40-40e4-9c13-d55c83c02e61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e3321d-ea40-40e4-9c13-d55c83c02e61-plugins-config-volume\") pod \"94e3321d-ea40-40e4-9c13-d55c83c02e61\" (UID: \"94e3321d-ea40-40e4-9c13-d55c83c02e61\") " Nov 01 23:08:54.723273 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:54.723048 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94e3321d-ea40-40e4-9c13-d55c83c02e61-kube-api-access-pmtsw\" (UniqueName: \"kubernetes.io/projected/94e3321d-ea40-40e4-9c13-d55c83c02e61-kube-api-access-pmtsw\") pod \"94e3321d-ea40-40e4-9c13-d55c83c02e61\" (UID: \"94e3321d-ea40-40e4-9c13-d55c83c02e61\") " Nov 01 23:08:54.723273 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:54.723103 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c828f45b-c1cd-4f21-8ab1-b050cba1a54e-kube-api-access-jqrzt\" (UniqueName: \"kubernetes.io/projected/c828f45b-c1cd-4f21-8ab1-b050cba1a54e-kube-api-access-jqrzt\") pod \"c828f45b-c1cd-4f21-8ab1-b050cba1a54e\" (UID: \"c828f45b-c1cd-4f21-8ab1-b050cba1a54e\") " Nov 01 23:08:54.723273 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:54.723160 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c828f45b-c1cd-4f21-8ab1-b050cba1a54e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c828f45b-c1cd-4f21-8ab1-b050cba1a54e-plugins-config-volume\") pod \"c828f45b-c1cd-4f21-8ab1-b050cba1a54e\" (UID: \"c828f45b-c1cd-4f21-8ab1-b050cba1a54e\") " Nov 01 23:08:54.724430 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:54.724398 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c828f45b-c1cd-4f21-8ab1-b050cba1a54e-plugins-config-volume" pod "c828f45b-c1cd-4f21-8ab1-b050cba1a54e" (UID: "c828f45b-c1cd-4f21-8ab1-b050cba1a54e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:08:54.726610 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:54.726574 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94e3321d-ea40-40e4-9c13-d55c83c02e61-plugins-config-volume" pod "94e3321d-ea40-40e4-9c13-d55c83c02e61" (UID: "94e3321d-ea40-40e4-9c13-d55c83c02e61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:08:54.729118 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:54.729075 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c828f45b-c1cd-4f21-8ab1-b050cba1a54e-kube-api-access-jqrzt" pod "c828f45b-c1cd-4f21-8ab1-b050cba1a54e" (UID: "c828f45b-c1cd-4f21-8ab1-b050cba1a54e"). InnerVolumeSpecName "kube-api-access-jqrzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:08:54.731953 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:54.731925 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e3321d-ea40-40e4-9c13-d55c83c02e61-kube-api-access-pmtsw" pod "94e3321d-ea40-40e4-9c13-d55c83c02e61" (UID: "94e3321d-ea40-40e4-9c13-d55c83c02e61"). InnerVolumeSpecName "kube-api-access-pmtsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:08:54.826254 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:54.826217 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e3321d-ea40-40e4-9c13-d55c83c02e61-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:54.826254 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:54.826254 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmtsw\" (UniqueName: \"kubernetes.io/projected/94e3321d-ea40-40e4-9c13-d55c83c02e61-kube-api-access-pmtsw\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:54.826458 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:54.826270 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqrzt\" (UniqueName: \"kubernetes.io/projected/c828f45b-c1cd-4f21-8ab1-b050cba1a54e-kube-api-access-jqrzt\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:54.826458 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:54.826285 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c828f45b-c1cd-4f21-8ab1-b050cba1a54e-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:55.094329 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:55.094296 5011 generic.go:356] "Generic (PLEG): container finished" podID="94e3321d-ea40-40e4-9c13-d55c83c02e61" containerID="1e851d4e1015415514f5223640bb8bc9be71f80b21450401c4f147409cf92d10" exitCode=0 Nov 01 23:08:55.094871 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:55.094353 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-49ab43-696d67bfd7-mkrrg" Nov 01 23:08:55.094871 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:55.094387 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-49ab43-696d67bfd7-mkrrg" event={"ID":"94e3321d-ea40-40e4-9c13-d55c83c02e61","Type":"ContainerDied","Data":"1e851d4e1015415514f5223640bb8bc9be71f80b21450401c4f147409cf92d10"} Nov 01 23:08:55.094871 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:55.094456 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-49ab43-696d67bfd7-mkrrg" event={"ID":"94e3321d-ea40-40e4-9c13-d55c83c02e61","Type":"ContainerDied","Data":"6a38bd44a40f5a773da1cb4ad0f7e323530a19848d070a982ce66cb607e4c5ab"} Nov 01 23:08:55.094871 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:55.094478 5011 scope.go:122] "RemoveContainer" containerID="1e851d4e1015415514f5223640bb8bc9be71f80b21450401c4f147409cf92d10" Nov 01 23:08:55.097449 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:55.097348 5011 generic.go:356] "Generic (PLEG): container finished" podID="c828f45b-c1cd-4f21-8ab1-b050cba1a54e" containerID="a72ec049be112cb0903f4f3108f301af12db7222bf7461520215faafd447c212" exitCode=0 Nov 01 23:08:55.097632 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:55.097604 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-49ab43-5bc95ddff9-m8bzq" event={"ID":"c828f45b-c1cd-4f21-8ab1-b050cba1a54e","Type":"ContainerDied","Data":"a72ec049be112cb0903f4f3108f301af12db7222bf7461520215faafd447c212"} Nov 01 23:08:55.097719 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:55.097667 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-49ab43-5bc95ddff9-m8bzq" event={"ID":"c828f45b-c1cd-4f21-8ab1-b050cba1a54e","Type":"ContainerDied","Data":"f072eae0faeb0a9f8f01a9bc3ae64d43eac097ce2c117618f6ef1d0cb296694f"} Nov 01 23:08:55.097776 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:55.097729 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-49ab43-5bc95ddff9-m8bzq" Nov 01 23:08:55.109405 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:55.109212 5011 scope.go:122] "RemoveContainer" containerID="1e851d4e1015415514f5223640bb8bc9be71f80b21450401c4f147409cf92d10" Nov 01 23:08:55.109525 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:08:55.109483 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e851d4e1015415514f5223640bb8bc9be71f80b21450401c4f147409cf92d10\": container with ID starting with 1e851d4e1015415514f5223640bb8bc9be71f80b21450401c4f147409cf92d10 not found: ID does not exist" containerID="1e851d4e1015415514f5223640bb8bc9be71f80b21450401c4f147409cf92d10" Nov 01 23:08:55.109599 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:55.109537 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e851d4e1015415514f5223640bb8bc9be71f80b21450401c4f147409cf92d10"} err="failed to get container status \"1e851d4e1015415514f5223640bb8bc9be71f80b21450401c4f147409cf92d10\": rpc error: code = NotFound desc = could not find container \"1e851d4e1015415514f5223640bb8bc9be71f80b21450401c4f147409cf92d10\": container with ID starting with 1e851d4e1015415514f5223640bb8bc9be71f80b21450401c4f147409cf92d10 not found: ID does not exist" Nov 01 23:08:55.109599 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:55.109560 5011 scope.go:122] "RemoveContainer" containerID="a72ec049be112cb0903f4f3108f301af12db7222bf7461520215faafd447c212" Nov 01 23:08:55.122660 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:55.122634 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-49ab43-696d67bfd7-mkrrg"] Nov 01 23:08:55.124188 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:55.124167 5011 scope.go:122] "RemoveContainer" containerID="a72ec049be112cb0903f4f3108f301af12db7222bf7461520215faafd447c212" Nov 01 23:08:55.124509 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:08:55.124485 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a72ec049be112cb0903f4f3108f301af12db7222bf7461520215faafd447c212\": container with ID starting with a72ec049be112cb0903f4f3108f301af12db7222bf7461520215faafd447c212 not found: ID does not exist" containerID="a72ec049be112cb0903f4f3108f301af12db7222bf7461520215faafd447c212" Nov 01 23:08:55.124589 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:55.124520 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a72ec049be112cb0903f4f3108f301af12db7222bf7461520215faafd447c212"} err="failed to get container status \"a72ec049be112cb0903f4f3108f301af12db7222bf7461520215faafd447c212\": rpc error: code = NotFound desc = could not find container \"a72ec049be112cb0903f4f3108f301af12db7222bf7461520215faafd447c212\": container with ID starting with a72ec049be112cb0903f4f3108f301af12db7222bf7461520215faafd447c212 not found: ID does not exist" Nov 01 23:08:55.129607 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:55.129583 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-49ab43-696d67bfd7-mkrrg"] Nov 01 23:08:55.140279 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:55.140254 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-49ab43-5bc95ddff9-m8bzq"] Nov 01 23:08:55.144313 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:55.144289 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-49ab43-5bc95ddff9-m8bzq"] Nov 01 23:08:55.750120 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:55.750089 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94e3321d-ea40-40e4-9c13-d55c83c02e61" path="/var/lib/kubelet/pods/94e3321d-ea40-40e4-9c13-d55c83c02e61/volumes" Nov 01 23:08:55.750591 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:55.750576 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c828f45b-c1cd-4f21-8ab1-b050cba1a54e" path="/var/lib/kubelet/pods/c828f45b-c1cd-4f21-8ab1-b050cba1a54e/volumes" Nov 01 23:08:55.751062 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:08:55.751046 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e06dee51-d25d-4e3c-bd5b-30ceb2bac53c" path="/var/lib/kubelet/pods/e06dee51-d25d-4e3c-bd5b-30ceb2bac53c/volumes" Nov 01 23:09:07.478887 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:07.478764 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-53dd05-fd7f899cc-tctb8"] Nov 01 23:09:07.513508 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:07.513455 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-53dd05-5b8b7fd495-b6jn6"] Nov 01 23:09:09.244478 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:09.244438 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:09:09.659160 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:09.658459 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt"] Nov 01 23:09:09.659160 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:09.658750 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt" podUID="92250b8d-b598-489f-a5d8-31a2d2ed954e" containerName="maas-api" containerID="cri-o://0f0f04d84a0fb7ac5fa88ffcaed4b747eb9d00d44eb2bca3179f29abce82eff1" gracePeriod=30 Nov 01 23:09:10.247774 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:10.246584 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt" Nov 01 23:09:10.323357 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:10.323322 5011 generic.go:356] "Generic (PLEG): container finished" podID="92250b8d-b598-489f-a5d8-31a2d2ed954e" containerID="0f0f04d84a0fb7ac5fa88ffcaed4b747eb9d00d44eb2bca3179f29abce82eff1" exitCode=0 Nov 01 23:09:10.323624 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:10.323384 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt" Nov 01 23:09:10.323624 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:10.323458 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt" event={"ID":"92250b8d-b598-489f-a5d8-31a2d2ed954e","Type":"ContainerDied","Data":"0f0f04d84a0fb7ac5fa88ffcaed4b747eb9d00d44eb2bca3179f29abce82eff1"} Nov 01 23:09:10.323624 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:10.323502 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt" event={"ID":"92250b8d-b598-489f-a5d8-31a2d2ed954e","Type":"ContainerDied","Data":"2ef8fdbb91100ff0c22c5eead3c5544cba0c49c69e650313df3b0f1448abad06"} Nov 01 23:09:10.323624 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:10.323524 5011 scope.go:122] "RemoveContainer" containerID="0f0f04d84a0fb7ac5fa88ffcaed4b747eb9d00d44eb2bca3179f29abce82eff1" Nov 01 23:09:10.342335 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:10.342307 5011 scope.go:122] "RemoveContainer" containerID="0f0f04d84a0fb7ac5fa88ffcaed4b747eb9d00d44eb2bca3179f29abce82eff1" Nov 01 23:09:10.342740 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:09:10.342703 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f0f04d84a0fb7ac5fa88ffcaed4b747eb9d00d44eb2bca3179f29abce82eff1\": container with ID starting with 0f0f04d84a0fb7ac5fa88ffcaed4b747eb9d00d44eb2bca3179f29abce82eff1 not found: ID does not exist" containerID="0f0f04d84a0fb7ac5fa88ffcaed4b747eb9d00d44eb2bca3179f29abce82eff1" Nov 01 23:09:10.342904 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:10.342750 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f0f04d84a0fb7ac5fa88ffcaed4b747eb9d00d44eb2bca3179f29abce82eff1"} err="failed to get container status \"0f0f04d84a0fb7ac5fa88ffcaed4b747eb9d00d44eb2bca3179f29abce82eff1\": rpc error: code = NotFound desc = could not find container \"0f0f04d84a0fb7ac5fa88ffcaed4b747eb9d00d44eb2bca3179f29abce82eff1\": container with ID starting with 0f0f04d84a0fb7ac5fa88ffcaed4b747eb9d00d44eb2bca3179f29abce82eff1 not found: ID does not exist" Nov 01 23:09:10.438957 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:10.438900 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92250b8d-b598-489f-a5d8-31a2d2ed954e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/92250b8d-b598-489f-a5d8-31a2d2ed954e-metrics-certs\") pod \"92250b8d-b598-489f-a5d8-31a2d2ed954e\" (UID: \"92250b8d-b598-489f-a5d8-31a2d2ed954e\") " Nov 01 23:09:10.439344 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:10.439311 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92250b8d-b598-489f-a5d8-31a2d2ed954e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92250b8d-b598-489f-a5d8-31a2d2ed954e-openshift-service-ca\") pod \"92250b8d-b598-489f-a5d8-31a2d2ed954e\" (UID: \"92250b8d-b598-489f-a5d8-31a2d2ed954e\") " Nov 01 23:09:10.439769 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:10.439742 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92250b8d-b598-489f-a5d8-31a2d2ed954e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92250b8d-b598-489f-a5d8-31a2d2ed954e-maas-api-tls\") pod \"92250b8d-b598-489f-a5d8-31a2d2ed954e\" (UID: \"92250b8d-b598-489f-a5d8-31a2d2ed954e\") " Nov 01 23:09:10.440014 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:10.439797 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92250b8d-b598-489f-a5d8-31a2d2ed954e-kube-api-access-2x46z\" (UniqueName: \"kubernetes.io/projected/92250b8d-b598-489f-a5d8-31a2d2ed954e-kube-api-access-2x46z\") pod \"92250b8d-b598-489f-a5d8-31a2d2ed954e\" (UID: \"92250b8d-b598-489f-a5d8-31a2d2ed954e\") " Nov 01 23:09:10.469472 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:10.467757 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92250b8d-b598-489f-a5d8-31a2d2ed954e-openshift-service-ca" pod "92250b8d-b598-489f-a5d8-31a2d2ed954e" (UID: "92250b8d-b598-489f-a5d8-31a2d2ed954e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:10.497474 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:10.497430 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92250b8d-b598-489f-a5d8-31a2d2ed954e-metrics-certs" pod "92250b8d-b598-489f-a5d8-31a2d2ed954e" (UID: "92250b8d-b598-489f-a5d8-31a2d2ed954e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:09:10.517824 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:10.499423 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92250b8d-b598-489f-a5d8-31a2d2ed954e-kube-api-access-2x46z" pod "92250b8d-b598-489f-a5d8-31a2d2ed954e" (UID: "92250b8d-b598-489f-a5d8-31a2d2ed954e"). InnerVolumeSpecName "kube-api-access-2x46z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:10.526011 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:10.525969 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92250b8d-b598-489f-a5d8-31a2d2ed954e-maas-api-tls" pod "92250b8d-b598-489f-a5d8-31a2d2ed954e" (UID: "92250b8d-b598-489f-a5d8-31a2d2ed954e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:09:10.558880 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:10.544219 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92250b8d-b598-489f-a5d8-31a2d2ed954e-metrics-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:10.558880 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:10.544257 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92250b8d-b598-489f-a5d8-31a2d2ed954e-openshift-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:10.558880 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:10.544272 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92250b8d-b598-489f-a5d8-31a2d2ed954e-maas-api-tls\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:10.558880 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:10.544286 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x46z\" (UniqueName: \"kubernetes.io/projected/92250b8d-b598-489f-a5d8-31a2d2ed954e-kube-api-access-2x46z\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:10.666938 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:10.666901 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt"] Nov 01 23:09:10.675088 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:10.675059 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-53dd05-7c8dd76864-9s8qt"] Nov 01 23:09:11.339758 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.338850 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245"] Nov 01 23:09:11.339758 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.339698 5011 cpu_manager.go:414] "containerMap: removing container" podUID="c828f45b-c1cd-4f21-8ab1-b050cba1a54e" containerName="payload-pre-processing" Nov 01 23:09:11.339758 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.339714 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c828f45b-c1cd-4f21-8ab1-b050cba1a54e" containerName="payload-pre-processing" Nov 01 23:09:11.339758 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.339734 5011 cpu_manager.go:414] "containerMap: removing container" podUID="92250b8d-b598-489f-a5d8-31a2d2ed954e" containerName="maas-api" Nov 01 23:09:11.339758 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.339742 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92250b8d-b598-489f-a5d8-31a2d2ed954e" containerName="maas-api" Nov 01 23:09:11.339758 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.339749 5011 cpu_manager.go:414] "containerMap: removing container" podUID="e06dee51-d25d-4e3c-bd5b-30ceb2bac53c" containerName="istio-proxy" Nov 01 23:09:11.339758 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.339754 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e06dee51-d25d-4e3c-bd5b-30ceb2bac53c" containerName="istio-proxy" Nov 01 23:09:11.340624 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.339786 5011 cpu_manager.go:414] "containerMap: removing container" podUID="c56973f4-5487-4aa0-88e9-c0e0161fb912" containerName="maas-api" Nov 01 23:09:11.340624 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.339791 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c56973f4-5487-4aa0-88e9-c0e0161fb912" containerName="maas-api" Nov 01 23:09:11.340624 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.339802 5011 cpu_manager.go:414] "containerMap: removing container" podUID="a6207c82-7f15-4c9a-9012-5c1aa1659dd0" containerName="payload-processing" Nov 01 23:09:11.340624 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.339809 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6207c82-7f15-4c9a-9012-5c1aa1659dd0" containerName="payload-processing" Nov 01 23:09:11.340624 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.339858 5011 cpu_manager.go:414] "containerMap: removing container" podUID="5e9a6cde-55d0-40d6-8fe7-e54452ef8da7" containerName="payload-pre-processing" Nov 01 23:09:11.340624 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.339866 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e9a6cde-55d0-40d6-8fe7-e54452ef8da7" containerName="payload-pre-processing" Nov 01 23:09:11.340624 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.339887 5011 cpu_manager.go:414] "containerMap: removing container" podUID="94e3321d-ea40-40e4-9c13-d55c83c02e61" containerName="payload-processing" Nov 01 23:09:11.340624 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.339894 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94e3321d-ea40-40e4-9c13-d55c83c02e61" containerName="payload-processing" Nov 01 23:09:11.340624 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.340073 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="c56973f4-5487-4aa0-88e9-c0e0161fb912" containerName="maas-api" Nov 01 23:09:11.340624 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.340086 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6207c82-7f15-4c9a-9012-5c1aa1659dd0" containerName="payload-processing" Nov 01 23:09:11.340624 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.340094 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e9a6cde-55d0-40d6-8fe7-e54452ef8da7" containerName="payload-pre-processing" Nov 01 23:09:11.340624 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.340101 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="94e3321d-ea40-40e4-9c13-d55c83c02e61" containerName="payload-processing" Nov 01 23:09:11.340624 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.340107 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="e06dee51-d25d-4e3c-bd5b-30ceb2bac53c" containerName="istio-proxy" Nov 01 23:09:11.340624 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.340120 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="c828f45b-c1cd-4f21-8ab1-b050cba1a54e" containerName="payload-pre-processing" Nov 01 23:09:11.340624 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.340129 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="92250b8d-b598-489f-a5d8-31a2d2ed954e" containerName="maas-api" Nov 01 23:09:11.340624 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.340134 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="e06dee51-d25d-4e3c-bd5b-30ceb2bac53c" containerName="istio-proxy" Nov 01 23:09:11.351796 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.351763 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:11.364637 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.364607 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-46ce02-openshift-default-dockercfg-s5dx6\"" Nov 01 23:09:11.386979 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.386939 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245"] Nov 01 23:09:11.474033 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.473996 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-istio-data\") pod \"e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:11.474203 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.474147 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-istio-envoy\") pod \"e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:11.474327 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.474308 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-credential-socket\") pod \"e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:11.474387 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.474370 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-workload-certs\") pod \"e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:11.474439 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.474423 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d295266d-5d50-4aae-a0ee-d86513a36f34-istiod-ca-cert\") pod \"e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:11.474497 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.474481 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sqr8\" (UniqueName: \"kubernetes.io/projected/d295266d-5d50-4aae-a0ee-d86513a36f34-kube-api-access-2sqr8\") pod \"e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:11.474569 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.474550 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d295266d-5d50-4aae-a0ee-d86513a36f34-istio-token\") pod \"e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:11.474606 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.474594 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-workload-socket\") pod \"e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:11.474679 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.474664 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d295266d-5d50-4aae-a0ee-d86513a36f34-istio-podinfo\") pod \"e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:11.577883 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.577816 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-istio-data\") pod \"e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:11.578088 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.577966 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-istio-envoy\") pod \"e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:11.578175 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.578083 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-credential-socket\") pod \"e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:11.578175 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.578157 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-workload-certs\") pod \"e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:11.578339 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.578181 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d295266d-5d50-4aae-a0ee-d86513a36f34-istiod-ca-cert\") pod \"e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:11.578339 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.578246 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sqr8\" (UniqueName: \"kubernetes.io/projected/d295266d-5d50-4aae-a0ee-d86513a36f34-kube-api-access-2sqr8\") pod \"e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:11.578339 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.578313 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d295266d-5d50-4aae-a0ee-d86513a36f34-istio-token\") pod \"e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:11.578500 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.578352 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-workload-socket\") pod \"e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:11.578500 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.578406 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d295266d-5d50-4aae-a0ee-d86513a36f34-istio-podinfo\") pod \"e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:11.579418 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.579386 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-workload-certs\") pod \"e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:11.579672 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.579652 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-istio-data\") pod \"e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:11.581031 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.581004 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d295266d-5d50-4aae-a0ee-d86513a36f34-istiod-ca-cert\") pod \"e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:11.581928 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.581798 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-credential-socket\") pod \"e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:11.585654 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.585630 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-workload-socket\") pod \"e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:11.586633 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.586594 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-istio-envoy\") pod \"e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:11.610427 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.602849 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d295266d-5d50-4aae-a0ee-d86513a36f34-istio-podinfo\") pod \"e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:11.610427 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.602916 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sqr8\" (UniqueName: \"kubernetes.io/projected/d295266d-5d50-4aae-a0ee-d86513a36f34-kube-api-access-2sqr8\") pod \"e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:11.610427 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.605005 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d295266d-5d50-4aae-a0ee-d86513a36f34-istio-token\") pod \"e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:11.680723 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.680684 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:11.764879 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:11.760133 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92250b8d-b598-489f-a5d8-31a2d2ed954e" path="/var/lib/kubelet/pods/92250b8d-b598-489f-a5d8-31a2d2ed954e/volumes" Nov 01 23:09:12.224262 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:12.224155 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245"] Nov 01 23:09:12.227025 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:09:12.226993 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd295266d_5d50_4aae_a0ee_d86513a36f34.slice/crio-df92e351f2882cf1b28396c8bb8120d51a9ac9d4113b0d8e8dee134088a28264 WatchSource:0}: Error finding container df92e351f2882cf1b28396c8bb8120d51a9ac9d4113b0d8e8dee134088a28264: Status 404 returned error can't find the container with id df92e351f2882cf1b28396c8bb8120d51a9ac9d4113b0d8e8dee134088a28264 Nov 01 23:09:12.236188 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:12.236139 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:09:12.236352 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:12.236236 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:09:12.236352 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:12.236284 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:09:12.271496 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:12.271444 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw"] Nov 01 23:09:12.273082 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:12.273042 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" podUID="5913d691-0d87-4599-b176-d132888e187e" containerName="istio-proxy" containerID="cri-o://b4167522c4317755f974ae58b3e915c0f4775a3c9bce3630c0b5d6b0546e2fb4" gracePeriod=30 Nov 01 23:09:12.373716 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:12.373671 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" event={"ID":"d295266d-5d50-4aae-a0ee-d86513a36f34","Type":"ContainerStarted","Data":"88d917aa5db9d3054d9536a28d70c0834b7e5c53af461700d1c6f3fcce812380"} Nov 01 23:09:12.374147 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:12.373746 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" event={"ID":"d295266d-5d50-4aae-a0ee-d86513a36f34","Type":"ContainerStarted","Data":"df92e351f2882cf1b28396c8bb8120d51a9ac9d4113b0d8e8dee134088a28264"} Nov 01 23:09:12.406436 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:12.406374 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" podStartSLOduration=1.406353411 podStartE2EDuration="1.406353411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:09:12.406082383 +0000 UTC m=+2720.810673015" watchObservedRunningTime="2026-11-01 23:09:12.406353411 +0000 UTC m=+2720.810944047" Nov 01 23:09:12.481051 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:12.480960 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-53dd05-fd7f899cc-tctb8" podUID="2c1fbfd9-c189-4b15-92aa-394d88d95ad8" containerName="payload-processing" containerID="cri-o://51c3422849600f7161e0bc6dc740a78a912a556bedd1e67aad934bd93a75db08" gracePeriod=25 Nov 01 23:09:12.516700 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:12.516657 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-53dd05-5b8b7fd495-b6jn6" podUID="5dbf96e5-22cf-4e5b-8208-470f9c1f616c" containerName="payload-pre-processing" containerID="cri-o://98e61759105f34f7a88c036dd865de0a554250924e18b91157cef2f40254f950" gracePeriod=25 Nov 01 23:09:12.681499 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:12.681462 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:13.705857 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:13.703119 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:14.390669 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:14.390634 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:14.391706 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:14.391682 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:09:16.984545 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:09:16.984503 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:09:17.416241 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:17.416202 5011 generic.go:356] "Generic (PLEG): container finished" podID="5913d691-0d87-4599-b176-d132888e187e" containerID="b4167522c4317755f974ae58b3e915c0f4775a3c9bce3630c0b5d6b0546e2fb4" exitCode=0 Nov 01 23:09:17.416441 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:17.416274 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" event={"ID":"5913d691-0d87-4599-b176-d132888e187e","Type":"ContainerDied","Data":"b4167522c4317755f974ae58b3e915c0f4775a3c9bce3630c0b5d6b0546e2fb4"} Nov 01 23:09:17.431606 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:17.431564 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c"] Nov 01 23:09:17.572790 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:17.572769 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 23:09:17.686261 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:17.686232 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-istio-envoy\") pod \"5913d691-0d87-4599-b176-d132888e187e\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " Nov 01 23:09:17.686437 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:17.686308 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5913d691-0d87-4599-b176-d132888e187e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5913d691-0d87-4599-b176-d132888e187e-istiod-ca-cert\") pod \"5913d691-0d87-4599-b176-d132888e187e\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " Nov 01 23:09:17.686437 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:17.686349 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-workload-socket\") pod \"5913d691-0d87-4599-b176-d132888e187e\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " Nov 01 23:09:17.686437 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:17.686416 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5913d691-0d87-4599-b176-d132888e187e-istio-token\" (UniqueName: \"kubernetes.io/projected/5913d691-0d87-4599-b176-d132888e187e-istio-token\") pod \"5913d691-0d87-4599-b176-d132888e187e\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " Nov 01 23:09:17.686579 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:17.686450 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-workload-certs\") pod \"5913d691-0d87-4599-b176-d132888e187e\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " Nov 01 23:09:17.686579 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:17.686518 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-credential-socket\") pod \"5913d691-0d87-4599-b176-d132888e187e\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " Nov 01 23:09:17.686579 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:17.686557 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5913d691-0d87-4599-b176-d132888e187e-kube-api-access-t4xtk\" (UniqueName: \"kubernetes.io/projected/5913d691-0d87-4599-b176-d132888e187e-kube-api-access-t4xtk\") pod \"5913d691-0d87-4599-b176-d132888e187e\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " Nov 01 23:09:17.686744 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:17.686691 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-workload-socket" pod "5913d691-0d87-4599-b176-d132888e187e" (UID: "5913d691-0d87-4599-b176-d132888e187e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:17.686744 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:17.686716 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-istio-data\") pod \"5913d691-0d87-4599-b176-d132888e187e\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " Nov 01 23:09:17.686869 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:17.686772 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-workload-certs" pod "5913d691-0d87-4599-b176-d132888e187e" (UID: "5913d691-0d87-4599-b176-d132888e187e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:17.688120 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:17.686799 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5913d691-0d87-4599-b176-d132888e187e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5913d691-0d87-4599-b176-d132888e187e-istio-podinfo\") pod \"5913d691-0d87-4599-b176-d132888e187e\" (UID: \"5913d691-0d87-4599-b176-d132888e187e\") " Nov 01 23:09:17.688120 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:17.686887 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-istio-data" pod "5913d691-0d87-4599-b176-d132888e187e" (UID: "5913d691-0d87-4599-b176-d132888e187e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:17.688120 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:17.687128 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-credential-socket" pod "5913d691-0d87-4599-b176-d132888e187e" (UID: "5913d691-0d87-4599-b176-d132888e187e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:17.688120 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:17.686988 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5913d691-0d87-4599-b176-d132888e187e-istiod-ca-cert" pod "5913d691-0d87-4599-b176-d132888e187e" (UID: "5913d691-0d87-4599-b176-d132888e187e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:17.688120 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:17.688024 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5913d691-0d87-4599-b176-d132888e187e-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:17.688120 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:17.688048 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:17.688120 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:17.688062 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:17.688120 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:17.688078 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:17.688120 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:17.688092 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:17.688710 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:17.688693 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-istio-envoy" pod "5913d691-0d87-4599-b176-d132888e187e" (UID: "5913d691-0d87-4599-b176-d132888e187e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:17.688850 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:17.688812 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5913d691-0d87-4599-b176-d132888e187e-istio-token" pod "5913d691-0d87-4599-b176-d132888e187e" (UID: "5913d691-0d87-4599-b176-d132888e187e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:17.688937 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:17.688917 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5913d691-0d87-4599-b176-d132888e187e-istio-podinfo" pod "5913d691-0d87-4599-b176-d132888e187e" (UID: "5913d691-0d87-4599-b176-d132888e187e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:09:17.689300 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:17.689279 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5913d691-0d87-4599-b176-d132888e187e-kube-api-access-t4xtk" pod "5913d691-0d87-4599-b176-d132888e187e" (UID: "5913d691-0d87-4599-b176-d132888e187e"). InnerVolumeSpecName "kube-api-access-t4xtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:17.789329 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:17.789301 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5913d691-0d87-4599-b176-d132888e187e-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:17.789329 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:17.789330 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5913d691-0d87-4599-b176-d132888e187e-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:17.789493 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:17.789340 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4xtk\" (UniqueName: \"kubernetes.io/projected/5913d691-0d87-4599-b176-d132888e187e-kube-api-access-t4xtk\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:17.789493 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:17.789349 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5913d691-0d87-4599-b176-d132888e187e-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:18.425066 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:18.425038 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" Nov 01 23:09:18.425539 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:18.425075 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw" event={"ID":"5913d691-0d87-4599-b176-d132888e187e","Type":"ContainerDied","Data":"0753b2bac246857d4110979814c7df068dd1ffa3d21e51fc37b28ea4878a4383"} Nov 01 23:09:18.425539 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:18.425149 5011 scope.go:122] "RemoveContainer" containerID="b4167522c4317755f974ae58b3e915c0f4775a3c9bce3630c0b5d6b0546e2fb4" Nov 01 23:09:18.445041 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:18.445010 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw"] Nov 01 23:09:18.455242 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:18.455209 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-q56sw"] Nov 01 23:09:19.748467 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:19.748432 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5913d691-0d87-4599-b176-d132888e187e" path="/var/lib/kubelet/pods/5913d691-0d87-4599-b176-d132888e187e/volumes" Nov 01 23:09:20.355110 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.355076 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z"] Nov 01 23:09:20.355988 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.355951 5011 cpu_manager.go:414] "containerMap: removing container" podUID="5913d691-0d87-4599-b176-d132888e187e" containerName="istio-proxy" Nov 01 23:09:20.355988 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.355975 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5913d691-0d87-4599-b176-d132888e187e" containerName="istio-proxy" Nov 01 23:09:20.356190 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.356044 5011 cpu_manager.go:414] "containerMap: removing container" podUID="e06dee51-d25d-4e3c-bd5b-30ceb2bac53c" containerName="istio-proxy" Nov 01 23:09:20.356190 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.356053 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e06dee51-d25d-4e3c-bd5b-30ceb2bac53c" containerName="istio-proxy" Nov 01 23:09:20.356303 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.356236 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="5913d691-0d87-4599-b176-d132888e187e" containerName="istio-proxy" Nov 01 23:09:20.362052 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.362017 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:20.383455 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.380333 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4"] Nov 01 23:09:20.389308 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.389281 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z"] Nov 01 23:09:20.389540 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.389519 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:20.395758 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.395728 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-01e30d-openshift-default-dockercfg-vsndd\"" Nov 01 23:09:20.401226 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.400862 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4"] Nov 01 23:09:20.447609 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.447573 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-credential-socket\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:20.447809 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.447753 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-workload-socket\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:20.447809 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.447802 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-workload-certs\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:20.447988 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.447965 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istio-envoy\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:20.448038 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.447999 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istio-podinfo\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:20.448078 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.448052 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istiod-ca-cert\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:20.448123 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.448099 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istio-token\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:20.448173 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.448139 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istio-data\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:20.448266 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.448252 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9d7w\" (UniqueName: \"kubernetes.io/projected/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-kube-api-access-b9d7w\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:20.549753 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.549718 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-workload-socket\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:20.549993 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.549771 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-workload-certs\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:20.549993 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.549797 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-workload-certs\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:20.549993 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.549814 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/824b3eb9-831a-4e29-a875-da055383cd7d-istio-token\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:20.549993 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.549972 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istio-envoy\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:20.550221 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.549998 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-credential-socket\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:20.550221 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.550028 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/824b3eb9-831a-4e29-a875-da055383cd7d-istiod-ca-cert\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:20.550221 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.550070 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istio-podinfo\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:20.550221 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.550142 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istiod-ca-cert\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:20.550221 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.550197 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-workload-socket\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:20.550221 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.550213 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-workload-socket\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:20.550462 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.550243 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istio-token\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:20.550462 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.550267 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-workload-certs\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:20.550462 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.550331 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istio-data\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:20.550462 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.550378 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7wdf\" (UniqueName: \"kubernetes.io/projected/824b3eb9-831a-4e29-a875-da055383cd7d-kube-api-access-n7wdf\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:20.550462 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.550422 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/824b3eb9-831a-4e29-a875-da055383cd7d-istio-podinfo\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:20.550462 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.550446 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9d7w\" (UniqueName: \"kubernetes.io/projected/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-kube-api-access-b9d7w\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:20.550724 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.550620 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-credential-socket\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:20.550724 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.550658 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-istio-data\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:20.550724 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.550719 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-istio-envoy\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:20.550912 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.550889 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istio-data\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:20.551038 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.551017 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-credential-socket\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:20.551273 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.551256 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istiod-ca-cert\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:20.552383 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.552366 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istio-envoy\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:20.552657 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.552642 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istio-podinfo\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:20.560064 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.560042 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istio-token\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:20.560270 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.560249 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9d7w\" (UniqueName: \"kubernetes.io/projected/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-kube-api-access-b9d7w\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:20.652472 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.652386 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-workload-certs\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:20.652472 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.652423 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/824b3eb9-831a-4e29-a875-da055383cd7d-istio-token\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:20.652700 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.652622 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-credential-socket\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:20.652700 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.652648 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/824b3eb9-831a-4e29-a875-da055383cd7d-istiod-ca-cert\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:20.652700 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.652696 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-workload-socket\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:20.652894 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.652804 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7wdf\" (UniqueName: \"kubernetes.io/projected/824b3eb9-831a-4e29-a875-da055383cd7d-kube-api-access-n7wdf\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:20.652894 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.652888 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/824b3eb9-831a-4e29-a875-da055383cd7d-istio-podinfo\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:20.653021 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.652993 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-credential-socket\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:20.653021 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.652883 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-workload-certs\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:20.653128 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.653041 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-istio-data\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:20.653128 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.653097 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-workload-socket\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:20.653128 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.653101 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-istio-envoy\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:20.653360 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.653336 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-istio-data\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:20.653524 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.653409 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/824b3eb9-831a-4e29-a875-da055383cd7d-istiod-ca-cert\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:20.655372 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.655349 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-istio-envoy\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:20.655640 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.655622 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/824b3eb9-831a-4e29-a875-da055383cd7d-istio-podinfo\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:20.661113 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.661091 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/824b3eb9-831a-4e29-a875-da055383cd7d-istio-token\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:20.661273 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.661252 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7wdf\" (UniqueName: \"kubernetes.io/projected/824b3eb9-831a-4e29-a875-da055383cd7d-kube-api-access-n7wdf\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:20.698274 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.698239 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:20.715795 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.715765 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:20.873178 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.872813 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm"] Nov 01 23:09:20.873178 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.873101 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" podUID="a32d183e-1080-4cef-ab97-f31aa667ef5c" containerName="istio-proxy" containerID="cri-o://b7f936c0b489a526ca56a2758ae47e11f71ff3fbbfc7538a43523b13be5b198b" gracePeriod=30 Nov 01 23:09:20.881161 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.881129 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5"] Nov 01 23:09:20.881411 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:20.881388 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" podUID="bc7cb565-6bf1-400f-9a3d-f062af8270bd" containerName="istio-proxy" containerID="cri-o://6df3bb6dd4358e48f785522540e63f02db67a98bdd1a32f63b07b721946ef51c" gracePeriod=30 Nov 01 23:09:21.002059 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:21.002026 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z"] Nov 01 23:09:21.005413 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:21.005376 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:09:21.005529 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:21.005458 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:09:21.005529 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:21.005499 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:09:21.026733 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:21.026710 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4"] Nov 01 23:09:21.027970 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:09:21.027894 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod824b3eb9_831a_4e29_a875_da055383cd7d.slice/crio-5047c0f0ec649a619ef3f53974ef4b82b88f62501afcf7507314a54eeaeb5f2f WatchSource:0}: Error finding container 5047c0f0ec649a619ef3f53974ef4b82b88f62501afcf7507314a54eeaeb5f2f: Status 404 returned error can't find the container with id 5047c0f0ec649a619ef3f53974ef4b82b88f62501afcf7507314a54eeaeb5f2f Nov 01 23:09:21.034159 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:21.034116 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:09:21.034281 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:21.034213 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:09:21.034281 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:21.034258 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:09:21.465930 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:21.465756 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" event={"ID":"824b3eb9-831a-4e29-a875-da055383cd7d","Type":"ContainerStarted","Data":"3740ecf9d40bc15fc5bbc76119758e2e2c85c29612f5265cdbe9bd389e6f108c"} Nov 01 23:09:21.465930 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:21.465887 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" event={"ID":"824b3eb9-831a-4e29-a875-da055383cd7d","Type":"ContainerStarted","Data":"5047c0f0ec649a619ef3f53974ef4b82b88f62501afcf7507314a54eeaeb5f2f"} Nov 01 23:09:21.469947 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:21.469920 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" event={"ID":"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c","Type":"ContainerStarted","Data":"ed7c4b5015a66ec471e3da78dc0846c0769e799e8a69c3790f15498bb45c30b0"} Nov 01 23:09:21.470082 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:21.469967 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" event={"ID":"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c","Type":"ContainerStarted","Data":"d90cfaf7994fcb081e1c8830ea20fc130aaf06c2e04e55dcfcfe8cdc847aad4d"} Nov 01 23:09:21.496181 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:21.496071 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" podStartSLOduration=1.496051209 podStartE2EDuration="1.496051209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:09:21.491149862 +0000 UTC m=+2729.895740494" watchObservedRunningTime="2026-11-01 23:09:21.496051209 +0000 UTC m=+2729.900641842" Nov 01 23:09:21.522900 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:21.522811 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" podStartSLOduration=1.5227913370000001 podStartE2EDuration="1.522791337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:09:21.517810542 +0000 UTC m=+2729.922401176" watchObservedRunningTime="2026-11-01 23:09:21.522791337 +0000 UTC m=+2729.927381970" Nov 01 23:09:21.698374 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:21.698349 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:21.716288 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:21.716255 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:21.723538 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:21.723504 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-53dd05-5b8b7fd495-b6jn6" podUID="5dbf96e5-22cf-4e5b-8208-470f9c1f616c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:09:21.723884 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:21.723858 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-53dd05-fd7f899cc-tctb8" podUID="2c1fbfd9-c189-4b15-92aa-394d88d95ad8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:09:22.700766 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:22.700732 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 01 23:09:22.701250 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:22.700792 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" podUID="88b7dfae-6aa7-41bf-b479-fb5468f9cc9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 01 23:09:22.717166 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:22.717138 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Nov 01 23:09:22.717322 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:22.717186 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" podUID="824b3eb9-831a-4e29-a875-da055383cd7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Nov 01 23:09:23.410546 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:23.410434 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz"] Nov 01 23:09:23.419187 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:23.419141 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz"] Nov 01 23:09:23.701324 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:23.701281 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 01 23:09:23.701695 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:23.701345 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" podUID="88b7dfae-6aa7-41bf-b479-fb5468f9cc9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 01 23:09:23.716751 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:23.716710 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Nov 01 23:09:23.716932 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:23.716784 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" podUID="824b3eb9-831a-4e29-a875-da055383cd7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Nov 01 23:09:24.699637 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:24.699598 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 01 23:09:24.699862 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:24.699654 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" podUID="88b7dfae-6aa7-41bf-b479-fb5468f9cc9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 01 23:09:24.716329 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:24.716293 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Nov 01 23:09:24.716679 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:24.716349 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" podUID="824b3eb9-831a-4e29-a875-da055383cd7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Nov 01 23:09:25.699855 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:25.699664 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 01 23:09:25.699855 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:25.699743 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" podUID="88b7dfae-6aa7-41bf-b479-fb5468f9cc9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 01 23:09:25.716428 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:25.716399 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Nov 01 23:09:25.716788 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:25.716447 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" podUID="824b3eb9-831a-4e29-a875-da055383cd7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Nov 01 23:09:26.512602 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.511755 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:09:26.598853 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.593483 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-workload-certs\") pod \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " Nov 01 23:09:26.598853 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.593549 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istio-podinfo\") pod \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " Nov 01 23:09:26.598853 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.593620 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istio-token\" (UniqueName: \"kubernetes.io/projected/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istio-token\") pod \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " Nov 01 23:09:26.598853 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.593656 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-workload-socket\") pod \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " Nov 01 23:09:26.598853 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.593709 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istio-data\") pod \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " Nov 01 23:09:26.598853 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.593774 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc7cb565-6bf1-400f-9a3d-f062af8270bd-kube-api-access-v9s2w\" (UniqueName: \"kubernetes.io/projected/bc7cb565-6bf1-400f-9a3d-f062af8270bd-kube-api-access-v9s2w\") pod \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " Nov 01 23:09:26.598853 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.593846 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istio-envoy\") pod \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " Nov 01 23:09:26.598853 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.593920 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-credential-socket\") pod \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " Nov 01 23:09:26.598853 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.593992 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istiod-ca-cert\") pod \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\" (UID: \"bc7cb565-6bf1-400f-9a3d-f062af8270bd\") " Nov 01 23:09:26.598853 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.595192 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istiod-ca-cert" pod "bc7cb565-6bf1-400f-9a3d-f062af8270bd" (UID: "bc7cb565-6bf1-400f-9a3d-f062af8270bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:26.598853 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.595492 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-workload-certs" pod "bc7cb565-6bf1-400f-9a3d-f062af8270bd" (UID: "bc7cb565-6bf1-400f-9a3d-f062af8270bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:26.598853 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.597572 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-credential-socket" pod "bc7cb565-6bf1-400f-9a3d-f062af8270bd" (UID: "bc7cb565-6bf1-400f-9a3d-f062af8270bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:26.598853 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.598015 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-workload-socket" pod "bc7cb565-6bf1-400f-9a3d-f062af8270bd" (UID: "bc7cb565-6bf1-400f-9a3d-f062af8270bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:26.598853 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.598353 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istio-data" pod "bc7cb565-6bf1-400f-9a3d-f062af8270bd" (UID: "bc7cb565-6bf1-400f-9a3d-f062af8270bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:26.604459 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.604418 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7cb565-6bf1-400f-9a3d-f062af8270bd-kube-api-access-v9s2w" pod "bc7cb565-6bf1-400f-9a3d-f062af8270bd" (UID: "bc7cb565-6bf1-400f-9a3d-f062af8270bd"). InnerVolumeSpecName "kube-api-access-v9s2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:26.607331 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.607295 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istio-token" pod "bc7cb565-6bf1-400f-9a3d-f062af8270bd" (UID: "bc7cb565-6bf1-400f-9a3d-f062af8270bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:26.610979 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.610944 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istio-podinfo" pod "bc7cb565-6bf1-400f-9a3d-f062af8270bd" (UID: "bc7cb565-6bf1-400f-9a3d-f062af8270bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:09:26.616016 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.615977 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istio-envoy" pod "bc7cb565-6bf1-400f-9a3d-f062af8270bd" (UID: "bc7cb565-6bf1-400f-9a3d-f062af8270bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:26.631399 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.631037 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:09:26.706588 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.704420 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 01 23:09:26.706588 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.704496 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" podUID="88b7dfae-6aa7-41bf-b479-fb5468f9cc9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 01 23:09:26.706588 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.706384 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.706588 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.706414 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.706588 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.706428 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.706588 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.706442 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.706588 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.706456 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.706588 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.706470 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9s2w\" (UniqueName: \"kubernetes.io/projected/bc7cb565-6bf1-400f-9a3d-f062af8270bd-kube-api-access-v9s2w\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.706588 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.706483 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.706588 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.706497 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7cb565-6bf1-400f-9a3d-f062af8270bd-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.706588 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.706520 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc7cb565-6bf1-400f-9a3d-f062af8270bd-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.717657 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.717621 5011 generic.go:356] "Generic (PLEG): container finished" podID="a32d183e-1080-4cef-ab97-f31aa667ef5c" containerID="b7f936c0b489a526ca56a2758ae47e11f71ff3fbbfc7538a43523b13be5b198b" exitCode=0 Nov 01 23:09:26.718138 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.717943 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" Nov 01 23:09:26.718138 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.717943 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" event={"ID":"a32d183e-1080-4cef-ab97-f31aa667ef5c","Type":"ContainerDied","Data":"b7f936c0b489a526ca56a2758ae47e11f71ff3fbbfc7538a43523b13be5b198b"} Nov 01 23:09:26.718138 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.718062 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm" event={"ID":"a32d183e-1080-4cef-ab97-f31aa667ef5c","Type":"ContainerDied","Data":"0985bd22a9f314a1f9b1edd7c743e39a6174935247f9c4a32853c7f6124fbf9a"} Nov 01 23:09:26.718138 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.718091 5011 scope.go:122] "RemoveContainer" containerID="b7f936c0b489a526ca56a2758ae47e11f71ff3fbbfc7538a43523b13be5b198b" Nov 01 23:09:26.718999 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.718932 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Nov 01 23:09:26.719115 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.718993 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" podUID="824b3eb9-831a-4e29-a875-da055383cd7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Nov 01 23:09:26.721942 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.721918 5011 generic.go:356] "Generic (PLEG): container finished" podID="bc7cb565-6bf1-400f-9a3d-f062af8270bd" containerID="6df3bb6dd4358e48f785522540e63f02db67a98bdd1a32f63b07b721946ef51c" exitCode=0 Nov 01 23:09:26.722072 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.722037 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" event={"ID":"bc7cb565-6bf1-400f-9a3d-f062af8270bd","Type":"ContainerDied","Data":"6df3bb6dd4358e48f785522540e63f02db67a98bdd1a32f63b07b721946ef51c"} Nov 01 23:09:26.722158 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.722079 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" event={"ID":"bc7cb565-6bf1-400f-9a3d-f062af8270bd","Type":"ContainerDied","Data":"a1cb1c4195aca9df4cb31586d0b860f1b4e6cbfed96924ed9464369c9e40847c"} Nov 01 23:09:26.722239 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.722159 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5" Nov 01 23:09:26.732164 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.732136 5011 scope.go:122] "RemoveContainer" containerID="b7f936c0b489a526ca56a2758ae47e11f71ff3fbbfc7538a43523b13be5b198b" Nov 01 23:09:26.732489 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:09:26.732468 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7f936c0b489a526ca56a2758ae47e11f71ff3fbbfc7538a43523b13be5b198b\": container with ID starting with b7f936c0b489a526ca56a2758ae47e11f71ff3fbbfc7538a43523b13be5b198b not found: ID does not exist" containerID="b7f936c0b489a526ca56a2758ae47e11f71ff3fbbfc7538a43523b13be5b198b" Nov 01 23:09:26.732580 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.732500 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7f936c0b489a526ca56a2758ae47e11f71ff3fbbfc7538a43523b13be5b198b"} err="failed to get container status \"b7f936c0b489a526ca56a2758ae47e11f71ff3fbbfc7538a43523b13be5b198b\": rpc error: code = NotFound desc = could not find container \"b7f936c0b489a526ca56a2758ae47e11f71ff3fbbfc7538a43523b13be5b198b\": container with ID starting with b7f936c0b489a526ca56a2758ae47e11f71ff3fbbfc7538a43523b13be5b198b not found: ID does not exist" Nov 01 23:09:26.732580 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.732524 5011 scope.go:122] "RemoveContainer" containerID="6df3bb6dd4358e48f785522540e63f02db67a98bdd1a32f63b07b721946ef51c" Nov 01 23:09:26.800885 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.800172 5011 scope.go:122] "RemoveContainer" containerID="ffbe0a2f2c8528da3bf0856546f4e257f4225f6f4875da3182c647330c97d2dc" Nov 01 23:09:26.803926 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.803866 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5"] Nov 01 23:09:26.809151 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.807635 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a32d183e-1080-4cef-ab97-f31aa667ef5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a32d183e-1080-4cef-ab97-f31aa667ef5c-istio-podinfo\") pod \"a32d183e-1080-4cef-ab97-f31aa667ef5c\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " Nov 01 23:09:26.809151 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.807738 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-credential-socket\") pod \"a32d183e-1080-4cef-ab97-f31aa667ef5c\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " Nov 01 23:09:26.809151 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.807876 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-workload-certs\") pod \"a32d183e-1080-4cef-ab97-f31aa667ef5c\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " Nov 01 23:09:26.809151 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.807914 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a32d183e-1080-4cef-ab97-f31aa667ef5c-kube-api-access-94bsm\" (UniqueName: \"kubernetes.io/projected/a32d183e-1080-4cef-ab97-f31aa667ef5c-kube-api-access-94bsm\") pod \"a32d183e-1080-4cef-ab97-f31aa667ef5c\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " Nov 01 23:09:26.809151 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.807947 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-istio-data\") pod \"a32d183e-1080-4cef-ab97-f31aa667ef5c\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " Nov 01 23:09:26.809151 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.808008 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-credential-socket" pod "a32d183e-1080-4cef-ab97-f31aa667ef5c" (UID: "a32d183e-1080-4cef-ab97-f31aa667ef5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:26.809151 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.808124 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-workload-socket\") pod \"a32d183e-1080-4cef-ab97-f31aa667ef5c\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " Nov 01 23:09:26.809151 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.808136 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-workload-certs" pod "a32d183e-1080-4cef-ab97-f31aa667ef5c" (UID: "a32d183e-1080-4cef-ab97-f31aa667ef5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:26.809151 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.808186 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a32d183e-1080-4cef-ab97-f31aa667ef5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a32d183e-1080-4cef-ab97-f31aa667ef5c-istiod-ca-cert\") pod \"a32d183e-1080-4cef-ab97-f31aa667ef5c\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " Nov 01 23:09:26.809151 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.808231 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a32d183e-1080-4cef-ab97-f31aa667ef5c-istio-token\" (UniqueName: \"kubernetes.io/projected/a32d183e-1080-4cef-ab97-f31aa667ef5c-istio-token\") pod \"a32d183e-1080-4cef-ab97-f31aa667ef5c\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " Nov 01 23:09:26.809151 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.808265 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-istio-envoy\") pod \"a32d183e-1080-4cef-ab97-f31aa667ef5c\" (UID: \"a32d183e-1080-4cef-ab97-f31aa667ef5c\") " Nov 01 23:09:26.809151 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.808314 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-istio-data" pod "a32d183e-1080-4cef-ab97-f31aa667ef5c" (UID: "a32d183e-1080-4cef-ab97-f31aa667ef5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:26.809151 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.808378 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-workload-socket" pod "a32d183e-1080-4cef-ab97-f31aa667ef5c" (UID: "a32d183e-1080-4cef-ab97-f31aa667ef5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:26.809151 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.808582 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a32d183e-1080-4cef-ab97-f31aa667ef5c-istiod-ca-cert" pod "a32d183e-1080-4cef-ab97-f31aa667ef5c" (UID: "a32d183e-1080-4cef-ab97-f31aa667ef5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:26.809151 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.809060 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.809151 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.809083 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.809151 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.809100 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.809151 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.809114 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.809151 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.809129 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a32d183e-1080-4cef-ab97-f31aa667ef5c-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.810241 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.809636 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-9rzv5"] Nov 01 23:09:26.810587 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.810554 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a32d183e-1080-4cef-ab97-f31aa667ef5c-istio-podinfo" pod "a32d183e-1080-4cef-ab97-f31aa667ef5c" (UID: "a32d183e-1080-4cef-ab97-f31aa667ef5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:09:26.810683 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.810644 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a32d183e-1080-4cef-ab97-f31aa667ef5c-istio-token" pod "a32d183e-1080-4cef-ab97-f31aa667ef5c" (UID: "a32d183e-1080-4cef-ab97-f31aa667ef5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:26.811243 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.811203 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-istio-envoy" pod "a32d183e-1080-4cef-ab97-f31aa667ef5c" (UID: "a32d183e-1080-4cef-ab97-f31aa667ef5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:26.811336 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.811248 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a32d183e-1080-4cef-ab97-f31aa667ef5c-kube-api-access-94bsm" pod "a32d183e-1080-4cef-ab97-f31aa667ef5c" (UID: "a32d183e-1080-4cef-ab97-f31aa667ef5c"). InnerVolumeSpecName "kube-api-access-94bsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:26.815104 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.815086 5011 scope.go:122] "RemoveContainer" containerID="6df3bb6dd4358e48f785522540e63f02db67a98bdd1a32f63b07b721946ef51c" Nov 01 23:09:26.815426 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:09:26.815408 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6df3bb6dd4358e48f785522540e63f02db67a98bdd1a32f63b07b721946ef51c\": container with ID starting with 6df3bb6dd4358e48f785522540e63f02db67a98bdd1a32f63b07b721946ef51c not found: ID does not exist" containerID="6df3bb6dd4358e48f785522540e63f02db67a98bdd1a32f63b07b721946ef51c" Nov 01 23:09:26.815477 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.815439 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df3bb6dd4358e48f785522540e63f02db67a98bdd1a32f63b07b721946ef51c"} err="failed to get container status \"6df3bb6dd4358e48f785522540e63f02db67a98bdd1a32f63b07b721946ef51c\": rpc error: code = NotFound desc = could not find container \"6df3bb6dd4358e48f785522540e63f02db67a98bdd1a32f63b07b721946ef51c\": container with ID starting with 6df3bb6dd4358e48f785522540e63f02db67a98bdd1a32f63b07b721946ef51c not found: ID does not exist" Nov 01 23:09:26.815477 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.815460 5011 scope.go:122] "RemoveContainer" containerID="ffbe0a2f2c8528da3bf0856546f4e257f4225f6f4875da3182c647330c97d2dc" Nov 01 23:09:26.815735 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:09:26.815718 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffbe0a2f2c8528da3bf0856546f4e257f4225f6f4875da3182c647330c97d2dc\": container with ID starting with ffbe0a2f2c8528da3bf0856546f4e257f4225f6f4875da3182c647330c97d2dc not found: ID does not exist" containerID="ffbe0a2f2c8528da3bf0856546f4e257f4225f6f4875da3182c647330c97d2dc" Nov 01 23:09:26.815791 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.815742 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffbe0a2f2c8528da3bf0856546f4e257f4225f6f4875da3182c647330c97d2dc"} err="failed to get container status \"ffbe0a2f2c8528da3bf0856546f4e257f4225f6f4875da3182c647330c97d2dc\": rpc error: code = NotFound desc = could not find container \"ffbe0a2f2c8528da3bf0856546f4e257f4225f6f4875da3182c647330c97d2dc\": container with ID starting with ffbe0a2f2c8528da3bf0856546f4e257f4225f6f4875da3182c647330c97d2dc not found: ID does not exist" Nov 01 23:09:26.910773 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.910737 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94bsm\" (UniqueName: \"kubernetes.io/projected/a32d183e-1080-4cef-ab97-f31aa667ef5c-kube-api-access-94bsm\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.910773 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.910771 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a32d183e-1080-4cef-ab97-f31aa667ef5c-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.910990 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.910788 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a32d183e-1080-4cef-ab97-f31aa667ef5c-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.910990 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:26.910804 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a32d183e-1080-4cef-ab97-f31aa667ef5c-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:27.086022 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:27.085950 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm"] Nov 01 23:09:27.099195 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:27.099156 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-rxnfm"] Nov 01 23:09:27.699394 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:27.699348 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 01 23:09:27.699845 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:27.699786 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" podUID="88b7dfae-6aa7-41bf-b479-fb5468f9cc9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 01 23:09:27.726163 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:27.726123 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:27.727619 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:27.727535 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:27.728181 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:27.728160 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:09:27.751989 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:27.751618 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a32d183e-1080-4cef-ab97-f31aa667ef5c" path="/var/lib/kubelet/pods/a32d183e-1080-4cef-ab97-f31aa667ef5c/volumes" Nov 01 23:09:27.755612 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:27.752515 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc7cb565-6bf1-400f-9a3d-f062af8270bd" path="/var/lib/kubelet/pods/bc7cb565-6bf1-400f-9a3d-f062af8270bd/volumes" Nov 01 23:09:28.707857 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:28.707720 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:28.710892 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:28.708409 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:28.710892 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:28.709095 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:09:31.638741 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:31.638700 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-788d7b-57874489c5-kkxpl"] Nov 01 23:09:31.727504 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:31.725674 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-53dd05-fd7f899cc-tctb8" podUID="2c1fbfd9-c189-4b15-92aa-394d88d95ad8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:09:31.727504 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:31.727451 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-53dd05-5b8b7fd495-b6jn6" podUID="5dbf96e5-22cf-4e5b-8208-470f9c1f616c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:09:32.484296 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:32.484248 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c" podUID="6f002a7d-fa04-46d4-a4ca-4df428b7e180" containerName="main" containerID="cri-o://463bb06caca02cfc34c951360d13e0781c74e8479d3e23af9c88f57ae091d9dc" gracePeriod=45 Nov 01 23:09:32.821282 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:32.821184 5011 generic.go:356] "Generic (PLEG): container finished" podID="6f002a7d-fa04-46d4-a4ca-4df428b7e180" containerID="463bb06caca02cfc34c951360d13e0781c74e8479d3e23af9c88f57ae091d9dc" exitCode=0 Nov 01 23:09:32.821710 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:32.821437 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c" event={"ID":"6f002a7d-fa04-46d4-a4ca-4df428b7e180","Type":"ContainerDied","Data":"463bb06caca02cfc34c951360d13e0781c74e8479d3e23af9c88f57ae091d9dc"} Nov 01 23:09:32.823476 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:32.823448 5011 generic.go:356] "Generic (PLEG): container finished" podID="5dbf96e5-22cf-4e5b-8208-470f9c1f616c" containerID="98e61759105f34f7a88c036dd865de0a554250924e18b91157cef2f40254f950" exitCode=0 Nov 01 23:09:32.823806 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:32.823779 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-53dd05-5b8b7fd495-b6jn6" event={"ID":"5dbf96e5-22cf-4e5b-8208-470f9c1f616c","Type":"ContainerDied","Data":"98e61759105f34f7a88c036dd865de0a554250924e18b91157cef2f40254f950"} Nov 01 23:09:32.832265 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:32.832238 5011 generic.go:356] "Generic (PLEG): container finished" podID="2c1fbfd9-c189-4b15-92aa-394d88d95ad8" containerID="51c3422849600f7161e0bc6dc740a78a912a556bedd1e67aad934bd93a75db08" exitCode=0 Nov 01 23:09:32.832512 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:32.832490 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-53dd05-fd7f899cc-tctb8" event={"ID":"2c1fbfd9-c189-4b15-92aa-394d88d95ad8","Type":"ContainerDied","Data":"51c3422849600f7161e0bc6dc740a78a912a556bedd1e67aad934bd93a75db08"} Nov 01 23:09:32.839620 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:32.839598 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-53dd05-fd7f899cc-tctb8" Nov 01 23:09:32.869234 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:32.869202 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c" Nov 01 23:09:32.915262 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:32.915167 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-53dd05-5b8b7fd495-b6jn6" Nov 01 23:09:32.988185 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:32.988150 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c1fbfd9-c189-4b15-92aa-394d88d95ad8-kube-api-access-z9tq4\" (UniqueName: \"kubernetes.io/projected/2c1fbfd9-c189-4b15-92aa-394d88d95ad8-kube-api-access-z9tq4\") pod \"2c1fbfd9-c189-4b15-92aa-394d88d95ad8\" (UID: \"2c1fbfd9-c189-4b15-92aa-394d88d95ad8\") " Nov 01 23:09:32.988395 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:32.988220 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f002a7d-fa04-46d4-a4ca-4df428b7e180-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f002a7d-fa04-46d4-a4ca-4df428b7e180-model-cache\") pod \"6f002a7d-fa04-46d4-a4ca-4df428b7e180\" (UID: \"6f002a7d-fa04-46d4-a4ca-4df428b7e180\") " Nov 01 23:09:32.988395 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:32.988317 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dbf96e5-22cf-4e5b-8208-470f9c1f616c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dbf96e5-22cf-4e5b-8208-470f9c1f616c-plugins-config-volume\") pod \"5dbf96e5-22cf-4e5b-8208-470f9c1f616c\" (UID: \"5dbf96e5-22cf-4e5b-8208-470f9c1f616c\") " Nov 01 23:09:32.988395 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:32.988343 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c1fbfd9-c189-4b15-92aa-394d88d95ad8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c1fbfd9-c189-4b15-92aa-394d88d95ad8-plugins-config-volume\") pod \"2c1fbfd9-c189-4b15-92aa-394d88d95ad8\" (UID: \"2c1fbfd9-c189-4b15-92aa-394d88d95ad8\") " Nov 01 23:09:32.988395 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:32.988390 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f002a7d-fa04-46d4-a4ca-4df428b7e180-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f002a7d-fa04-46d4-a4ca-4df428b7e180-dshm\") pod \"6f002a7d-fa04-46d4-a4ca-4df428b7e180\" (UID: \"6f002a7d-fa04-46d4-a4ca-4df428b7e180\") " Nov 01 23:09:32.988613 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:32.988486 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f002a7d-fa04-46d4-a4ca-4df428b7e180-kube-api-access-zs65d\" (UniqueName: \"kubernetes.io/projected/6f002a7d-fa04-46d4-a4ca-4df428b7e180-kube-api-access-zs65d\") pod \"6f002a7d-fa04-46d4-a4ca-4df428b7e180\" (UID: \"6f002a7d-fa04-46d4-a4ca-4df428b7e180\") " Nov 01 23:09:32.988613 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:32.988562 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f002a7d-fa04-46d4-a4ca-4df428b7e180-home\" (UniqueName: \"kubernetes.io/empty-dir/6f002a7d-fa04-46d4-a4ca-4df428b7e180-home\") pod \"6f002a7d-fa04-46d4-a4ca-4df428b7e180\" (UID: \"6f002a7d-fa04-46d4-a4ca-4df428b7e180\") " Nov 01 23:09:32.988613 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:32.988608 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f002a7d-fa04-46d4-a4ca-4df428b7e180-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f002a7d-fa04-46d4-a4ca-4df428b7e180-tmp-dir\") pod \"6f002a7d-fa04-46d4-a4ca-4df428b7e180\" (UID: \"6f002a7d-fa04-46d4-a4ca-4df428b7e180\") " Nov 01 23:09:32.988771 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:32.988682 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f002a7d-fa04-46d4-a4ca-4df428b7e180-tls-certs\" (UniqueName: \"kubernetes.io/secret/6f002a7d-fa04-46d4-a4ca-4df428b7e180-tls-certs\") pod \"6f002a7d-fa04-46d4-a4ca-4df428b7e180\" (UID: \"6f002a7d-fa04-46d4-a4ca-4df428b7e180\") " Nov 01 23:09:32.988823 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:32.988785 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dbf96e5-22cf-4e5b-8208-470f9c1f616c-kube-api-access-4d5nj\" (UniqueName: \"kubernetes.io/projected/5dbf96e5-22cf-4e5b-8208-470f9c1f616c-kube-api-access-4d5nj\") pod \"5dbf96e5-22cf-4e5b-8208-470f9c1f616c\" (UID: \"5dbf96e5-22cf-4e5b-8208-470f9c1f616c\") " Nov 01 23:09:32.988823 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:32.988804 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dbf96e5-22cf-4e5b-8208-470f9c1f616c-plugins-config-volume" pod "5dbf96e5-22cf-4e5b-8208-470f9c1f616c" (UID: "5dbf96e5-22cf-4e5b-8208-470f9c1f616c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:32.989124 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:32.989102 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f002a7d-fa04-46d4-a4ca-4df428b7e180-model-cache" pod "6f002a7d-fa04-46d4-a4ca-4df428b7e180" (UID: "6f002a7d-fa04-46d4-a4ca-4df428b7e180"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:32.989488 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:32.989465 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f002a7d-fa04-46d4-a4ca-4df428b7e180-model-cache\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:32.989647 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:32.989494 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dbf96e5-22cf-4e5b-8208-470f9c1f616c-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:32.990219 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:32.990183 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c1fbfd9-c189-4b15-92aa-394d88d95ad8-plugins-config-volume" pod "2c1fbfd9-c189-4b15-92aa-394d88d95ad8" (UID: "2c1fbfd9-c189-4b15-92aa-394d88d95ad8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:32.990494 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:32.990447 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f002a7d-fa04-46d4-a4ca-4df428b7e180-tmp-dir" pod "6f002a7d-fa04-46d4-a4ca-4df428b7e180" (UID: "6f002a7d-fa04-46d4-a4ca-4df428b7e180"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:32.990707 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:32.990639 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f002a7d-fa04-46d4-a4ca-4df428b7e180-home" pod "6f002a7d-fa04-46d4-a4ca-4df428b7e180" (UID: "6f002a7d-fa04-46d4-a4ca-4df428b7e180"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:32.992576 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:32.992554 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dbf96e5-22cf-4e5b-8208-470f9c1f616c-kube-api-access-4d5nj" pod "5dbf96e5-22cf-4e5b-8208-470f9c1f616c" (UID: "5dbf96e5-22cf-4e5b-8208-470f9c1f616c"). InnerVolumeSpecName "kube-api-access-4d5nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:32.992857 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:32.992761 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f002a7d-fa04-46d4-a4ca-4df428b7e180-kube-api-access-zs65d" pod "6f002a7d-fa04-46d4-a4ca-4df428b7e180" (UID: "6f002a7d-fa04-46d4-a4ca-4df428b7e180"). InnerVolumeSpecName "kube-api-access-zs65d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:32.993174 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:32.993146 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c1fbfd9-c189-4b15-92aa-394d88d95ad8-kube-api-access-z9tq4" pod "2c1fbfd9-c189-4b15-92aa-394d88d95ad8" (UID: "2c1fbfd9-c189-4b15-92aa-394d88d95ad8"). InnerVolumeSpecName "kube-api-access-z9tq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:32.993645 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:32.993624 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f002a7d-fa04-46d4-a4ca-4df428b7e180-tls-certs" pod "6f002a7d-fa04-46d4-a4ca-4df428b7e180" (UID: "6f002a7d-fa04-46d4-a4ca-4df428b7e180"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:09:33.012861 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:33.005496 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f002a7d-fa04-46d4-a4ca-4df428b7e180-dshm" pod "6f002a7d-fa04-46d4-a4ca-4df428b7e180" (UID: "6f002a7d-fa04-46d4-a4ca-4df428b7e180"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:33.090442 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:33.090354 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs65d\" (UniqueName: \"kubernetes.io/projected/6f002a7d-fa04-46d4-a4ca-4df428b7e180-kube-api-access-zs65d\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:33.090442 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:33.090383 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f002a7d-fa04-46d4-a4ca-4df428b7e180-home\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:33.090442 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:33.090392 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f002a7d-fa04-46d4-a4ca-4df428b7e180-tmp-dir\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:33.090442 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:33.090400 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f002a7d-fa04-46d4-a4ca-4df428b7e180-tls-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:33.090442 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:33.090410 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d5nj\" (UniqueName: \"kubernetes.io/projected/5dbf96e5-22cf-4e5b-8208-470f9c1f616c-kube-api-access-4d5nj\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:33.090442 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:33.090418 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9tq4\" (UniqueName: \"kubernetes.io/projected/2c1fbfd9-c189-4b15-92aa-394d88d95ad8-kube-api-access-z9tq4\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:33.090442 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:33.090426 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c1fbfd9-c189-4b15-92aa-394d88d95ad8-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:33.090442 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:33.090435 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f002a7d-fa04-46d4-a4ca-4df428b7e180-dshm\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:33.841036 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:33.841007 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-53dd05-5b8b7fd495-b6jn6" Nov 01 23:09:33.841562 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:33.841039 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-53dd05-5b8b7fd495-b6jn6" event={"ID":"5dbf96e5-22cf-4e5b-8208-470f9c1f616c","Type":"ContainerDied","Data":"c94e168859cb936f18ebc57bc093b65856762ce04683714233ab17a896e042ec"} Nov 01 23:09:33.841562 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:33.841100 5011 scope.go:122] "RemoveContainer" containerID="98e61759105f34f7a88c036dd865de0a554250924e18b91157cef2f40254f950" Nov 01 23:09:33.843852 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:33.843796 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-53dd05-fd7f899cc-tctb8" Nov 01 23:09:33.844024 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:33.843979 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-53dd05-fd7f899cc-tctb8" event={"ID":"2c1fbfd9-c189-4b15-92aa-394d88d95ad8","Type":"ContainerDied","Data":"6310721042c813c4b126e7c1dbd5fe7f9d7e75bb20e9c1cda5d56cd6f9119ed9"} Nov 01 23:09:33.846794 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:33.846760 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c" event={"ID":"6f002a7d-fa04-46d4-a4ca-4df428b7e180","Type":"ContainerDied","Data":"6b78a1373855e42128e3468adf8062b6ff18d19976a0d17184ced89a118d6927"} Nov 01 23:09:33.846959 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:33.846783 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c" Nov 01 23:09:33.855249 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:33.855225 5011 scope.go:122] "RemoveContainer" containerID="51c3422849600f7161e0bc6dc740a78a912a556bedd1e67aad934bd93a75db08" Nov 01 23:09:33.860195 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:33.860170 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-53dd05-5b8b7fd495-b6jn6"] Nov 01 23:09:33.864015 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:33.863993 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-53dd05-5b8b7fd495-b6jn6"] Nov 01 23:09:33.866052 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:33.866034 5011 scope.go:122] "RemoveContainer" containerID="463bb06caca02cfc34c951360d13e0781c74e8479d3e23af9c88f57ae091d9dc" Nov 01 23:09:33.881136 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:33.881116 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-53dd05-fd7f899cc-tctb8"] Nov 01 23:09:33.886747 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:33.886723 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-53dd05-fd7f899cc-tctb8"] Nov 01 23:09:33.898354 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:33.898329 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c"] Nov 01 23:09:33.902419 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:33.902398 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-337dee/ipp-route-337dee-kserve-658574fc9c-bbt6c"] Nov 01 23:09:35.769363 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:35.769326 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c1fbfd9-c189-4b15-92aa-394d88d95ad8" path="/var/lib/kubelet/pods/2c1fbfd9-c189-4b15-92aa-394d88d95ad8/volumes" Nov 01 23:09:35.769804 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:35.769789 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dbf96e5-22cf-4e5b-8208-470f9c1f616c" path="/var/lib/kubelet/pods/5dbf96e5-22cf-4e5b-8208-470f9c1f616c/volumes" Nov 01 23:09:35.770298 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:35.770284 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f002a7d-fa04-46d4-a4ca-4df428b7e180" path="/var/lib/kubelet/pods/6f002a7d-fa04-46d4-a4ca-4df428b7e180/volumes" Nov 01 23:09:36.639949 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:36.639888 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-788d7b-57874489c5-kkxpl" podUID="7e080376-e7a9-406c-b102-cc3cd7c5ebcb" containerName="payload-pre-processing" containerID="cri-o://526310d78ebbd50c674182729be237cedfd15e6c93b8942204a226d28c19d9db" gracePeriod=25 Nov 01 23:09:38.604300 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:38.604265 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" containerID="cri-o://4788ab542bd84628d8b3a1c6cbbfd4af93208e2341585140995d0fd03c14c5b5" gracePeriod=45 Nov 01 23:09:38.623238 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:38.623054 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" containerID="cri-o://69cff53ae35b42778eb93834c7ef391af268cc2bfcf4141b69cb3791576988f8" gracePeriod=45 Nov 01 23:09:38.892485 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:38.892394 5011 generic.go:356] "Generic (PLEG): container finished" podID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerID="4788ab542bd84628d8b3a1c6cbbfd4af93208e2341585140995d0fd03c14c5b5" exitCode=0 Nov 01 23:09:38.892816 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:38.892785 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" event={"ID":"51df36c5-b9ce-4b2f-8525-5aa2d8508801","Type":"ContainerDied","Data":"4788ab542bd84628d8b3a1c6cbbfd4af93208e2341585140995d0fd03c14c5b5"} Nov 01 23:09:38.894943 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:38.894918 5011 generic.go:356] "Generic (PLEG): container finished" podID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerID="69cff53ae35b42778eb93834c7ef391af268cc2bfcf4141b69cb3791576988f8" exitCode=0 Nov 01 23:09:38.895072 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:38.894995 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" event={"ID":"3508619c-ddff-4d19-8e29-a28cb562de3b","Type":"ContainerDied","Data":"69cff53ae35b42778eb93834c7ef391af268cc2bfcf4141b69cb3791576988f8"} Nov 01 23:09:39.024320 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.022200 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 23:09:39.026732 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.026711 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 23:09:39.106851 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.106796 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3508619c-ddff-4d19-8e29-a28cb562de3b-home\" (UniqueName: \"kubernetes.io/empty-dir/3508619c-ddff-4d19-8e29-a28cb562de3b-home\") pod \"3508619c-ddff-4d19-8e29-a28cb562de3b\" (UID: \"3508619c-ddff-4d19-8e29-a28cb562de3b\") " Nov 01 23:09:39.107026 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.106929 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51df36c5-b9ce-4b2f-8525-5aa2d8508801-dshm\" (UniqueName: \"kubernetes.io/empty-dir/51df36c5-b9ce-4b2f-8525-5aa2d8508801-dshm\") pod \"51df36c5-b9ce-4b2f-8525-5aa2d8508801\" (UID: \"51df36c5-b9ce-4b2f-8525-5aa2d8508801\") " Nov 01 23:09:39.107026 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.106977 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3508619c-ddff-4d19-8e29-a28cb562de3b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3508619c-ddff-4d19-8e29-a28cb562de3b-tmp-dir\") pod \"3508619c-ddff-4d19-8e29-a28cb562de3b\" (UID: \"3508619c-ddff-4d19-8e29-a28cb562de3b\") " Nov 01 23:09:39.107026 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.107021 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51df36c5-b9ce-4b2f-8525-5aa2d8508801-tls-certs\" (UniqueName: \"kubernetes.io/secret/51df36c5-b9ce-4b2f-8525-5aa2d8508801-tls-certs\") pod \"51df36c5-b9ce-4b2f-8525-5aa2d8508801\" (UID: \"51df36c5-b9ce-4b2f-8525-5aa2d8508801\") " Nov 01 23:09:39.107231 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.107061 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3508619c-ddff-4d19-8e29-a28cb562de3b-home" pod "3508619c-ddff-4d19-8e29-a28cb562de3b" (UID: "3508619c-ddff-4d19-8e29-a28cb562de3b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:39.107231 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.107083 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51df36c5-b9ce-4b2f-8525-5aa2d8508801-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51df36c5-b9ce-4b2f-8525-5aa2d8508801-model-cache\") pod \"51df36c5-b9ce-4b2f-8525-5aa2d8508801\" (UID: \"51df36c5-b9ce-4b2f-8525-5aa2d8508801\") " Nov 01 23:09:39.107231 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.107163 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3508619c-ddff-4d19-8e29-a28cb562de3b-tls-certs\" (UniqueName: \"kubernetes.io/secret/3508619c-ddff-4d19-8e29-a28cb562de3b-tls-certs\") pod \"3508619c-ddff-4d19-8e29-a28cb562de3b\" (UID: \"3508619c-ddff-4d19-8e29-a28cb562de3b\") " Nov 01 23:09:39.107231 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.107226 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51df36c5-b9ce-4b2f-8525-5aa2d8508801-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51df36c5-b9ce-4b2f-8525-5aa2d8508801-tmp-dir\") pod \"51df36c5-b9ce-4b2f-8525-5aa2d8508801\" (UID: \"51df36c5-b9ce-4b2f-8525-5aa2d8508801\") " Nov 01 23:09:39.107456 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.107260 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3508619c-ddff-4d19-8e29-a28cb562de3b-tmp-dir" pod "3508619c-ddff-4d19-8e29-a28cb562de3b" (UID: "3508619c-ddff-4d19-8e29-a28cb562de3b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:39.107456 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.107296 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51df36c5-b9ce-4b2f-8525-5aa2d8508801-home\" (UniqueName: \"kubernetes.io/empty-dir/51df36c5-b9ce-4b2f-8525-5aa2d8508801-home\") pod \"51df36c5-b9ce-4b2f-8525-5aa2d8508801\" (UID: \"51df36c5-b9ce-4b2f-8525-5aa2d8508801\") " Nov 01 23:09:39.107456 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.107327 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51df36c5-b9ce-4b2f-8525-5aa2d8508801-kube-api-access-kg9vk\" (UniqueName: \"kubernetes.io/projected/51df36c5-b9ce-4b2f-8525-5aa2d8508801-kube-api-access-kg9vk\") pod \"51df36c5-b9ce-4b2f-8525-5aa2d8508801\" (UID: \"51df36c5-b9ce-4b2f-8525-5aa2d8508801\") " Nov 01 23:09:39.107456 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.107401 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3508619c-ddff-4d19-8e29-a28cb562de3b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3508619c-ddff-4d19-8e29-a28cb562de3b-model-cache\") pod \"3508619c-ddff-4d19-8e29-a28cb562de3b\" (UID: \"3508619c-ddff-4d19-8e29-a28cb562de3b\") " Nov 01 23:09:39.107456 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.107427 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3508619c-ddff-4d19-8e29-a28cb562de3b-kube-api-access-mtnz5\" (UniqueName: \"kubernetes.io/projected/3508619c-ddff-4d19-8e29-a28cb562de3b-kube-api-access-mtnz5\") pod \"3508619c-ddff-4d19-8e29-a28cb562de3b\" (UID: \"3508619c-ddff-4d19-8e29-a28cb562de3b\") " Nov 01 23:09:39.107652 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.107490 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3508619c-ddff-4d19-8e29-a28cb562de3b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3508619c-ddff-4d19-8e29-a28cb562de3b-dshm\") pod \"3508619c-ddff-4d19-8e29-a28cb562de3b\" (UID: \"3508619c-ddff-4d19-8e29-a28cb562de3b\") " Nov 01 23:09:39.107739 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.107715 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51df36c5-b9ce-4b2f-8525-5aa2d8508801-tmp-dir" pod "51df36c5-b9ce-4b2f-8525-5aa2d8508801" (UID: "51df36c5-b9ce-4b2f-8525-5aa2d8508801"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:39.108113 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.108087 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3508619c-ddff-4d19-8e29-a28cb562de3b-model-cache" pod "3508619c-ddff-4d19-8e29-a28cb562de3b" (UID: "3508619c-ddff-4d19-8e29-a28cb562de3b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:39.108249 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.108237 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3508619c-ddff-4d19-8e29-a28cb562de3b-home\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:39.108319 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.108259 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3508619c-ddff-4d19-8e29-a28cb562de3b-tmp-dir\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:39.108319 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.108260 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51df36c5-b9ce-4b2f-8525-5aa2d8508801-home" pod "51df36c5-b9ce-4b2f-8525-5aa2d8508801" (UID: "51df36c5-b9ce-4b2f-8525-5aa2d8508801"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:39.108319 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.108272 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51df36c5-b9ce-4b2f-8525-5aa2d8508801-tmp-dir\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:39.108319 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.108286 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3508619c-ddff-4d19-8e29-a28cb562de3b-model-cache\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:39.108816 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.108783 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51df36c5-b9ce-4b2f-8525-5aa2d8508801-model-cache" pod "51df36c5-b9ce-4b2f-8525-5aa2d8508801" (UID: "51df36c5-b9ce-4b2f-8525-5aa2d8508801"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:39.110427 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.110399 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51df36c5-b9ce-4b2f-8525-5aa2d8508801-dshm" pod "51df36c5-b9ce-4b2f-8525-5aa2d8508801" (UID: "51df36c5-b9ce-4b2f-8525-5aa2d8508801"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:39.110427 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.110427 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51df36c5-b9ce-4b2f-8525-5aa2d8508801-tls-certs" pod "51df36c5-b9ce-4b2f-8525-5aa2d8508801" (UID: "51df36c5-b9ce-4b2f-8525-5aa2d8508801"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:09:39.111023 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.111002 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3508619c-ddff-4d19-8e29-a28cb562de3b-tls-certs" pod "3508619c-ddff-4d19-8e29-a28cb562de3b" (UID: "3508619c-ddff-4d19-8e29-a28cb562de3b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:09:39.111197 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.111174 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3508619c-ddff-4d19-8e29-a28cb562de3b-kube-api-access-mtnz5" pod "3508619c-ddff-4d19-8e29-a28cb562de3b" (UID: "3508619c-ddff-4d19-8e29-a28cb562de3b"). InnerVolumeSpecName "kube-api-access-mtnz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:39.112642 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.112622 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3508619c-ddff-4d19-8e29-a28cb562de3b-dshm" pod "3508619c-ddff-4d19-8e29-a28cb562de3b" (UID: "3508619c-ddff-4d19-8e29-a28cb562de3b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:39.112963 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.112944 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51df36c5-b9ce-4b2f-8525-5aa2d8508801-kube-api-access-kg9vk" pod "51df36c5-b9ce-4b2f-8525-5aa2d8508801" (UID: "51df36c5-b9ce-4b2f-8525-5aa2d8508801"). InnerVolumeSpecName "kube-api-access-kg9vk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:39.209712 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.209667 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51df36c5-b9ce-4b2f-8525-5aa2d8508801-dshm\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:39.209712 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.209704 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51df36c5-b9ce-4b2f-8525-5aa2d8508801-tls-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:39.209712 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.209716 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51df36c5-b9ce-4b2f-8525-5aa2d8508801-model-cache\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:39.210082 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.209724 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3508619c-ddff-4d19-8e29-a28cb562de3b-tls-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:39.210082 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.209733 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51df36c5-b9ce-4b2f-8525-5aa2d8508801-home\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:39.210082 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.209742 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg9vk\" (UniqueName: \"kubernetes.io/projected/51df36c5-b9ce-4b2f-8525-5aa2d8508801-kube-api-access-kg9vk\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:39.210082 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.209751 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtnz5\" (UniqueName: \"kubernetes.io/projected/3508619c-ddff-4d19-8e29-a28cb562de3b-kube-api-access-mtnz5\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:39.210082 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.209759 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3508619c-ddff-4d19-8e29-a28cb562de3b-dshm\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:39.403771 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.403731 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn"] Nov 01 23:09:39.404042 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.404014 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn" podUID="129162f7-0fdd-426d-a66f-6acecdf14172" containerName="maas-api" containerID="cri-o://e4a66c28b2d29ca75841b2c64a8b66873a0c56a8dcae8c67eff2bd14080e802e" gracePeriod=30 Nov 01 23:09:39.825006 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.824480 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Nov 01 23:09:39.862395 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.862361 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn" Nov 01 23:09:39.910593 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.910374 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" Nov 01 23:09:39.910593 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.910401 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz" event={"ID":"51df36c5-b9ce-4b2f-8525-5aa2d8508801","Type":"ContainerDied","Data":"b949a62095ead76c4fde339dd92ddcfd587976fb83f232368e60c44202e6122d"} Nov 01 23:09:39.910593 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.910460 5011 scope.go:122] "RemoveContainer" containerID="4788ab542bd84628d8b3a1c6cbbfd4af93208e2341585140995d0fd03c14c5b5" Nov 01 23:09:39.912538 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.912507 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" Nov 01 23:09:39.912664 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.912589 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" event={"ID":"3508619c-ddff-4d19-8e29-a28cb562de3b","Type":"ContainerDied","Data":"846b2260ac315d712b5c14b030c3be105c3cd76f53a78f998975b22881609052"} Nov 01 23:09:39.916089 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.914385 5011 generic.go:356] "Generic (PLEG): container finished" podID="129162f7-0fdd-426d-a66f-6acecdf14172" containerID="e4a66c28b2d29ca75841b2c64a8b66873a0c56a8dcae8c67eff2bd14080e802e" exitCode=0 Nov 01 23:09:39.916089 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.914581 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn" event={"ID":"129162f7-0fdd-426d-a66f-6acecdf14172","Type":"ContainerDied","Data":"e4a66c28b2d29ca75841b2c64a8b66873a0c56a8dcae8c67eff2bd14080e802e"} Nov 01 23:09:39.916089 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.914620 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn" event={"ID":"129162f7-0fdd-426d-a66f-6acecdf14172","Type":"ContainerDied","Data":"19690b948edc57c3c02e24216d4ad151dc14735b96e625edd8bea2e49d142c02"} Nov 01 23:09:39.916089 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.914689 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn" Nov 01 23:09:39.925133 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.925108 5011 scope.go:122] "RemoveContainer" containerID="69cff53ae35b42778eb93834c7ef391af268cc2bfcf4141b69cb3791576988f8" Nov 01 23:09:39.938321 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.938299 5011 scope.go:122] "RemoveContainer" containerID="e4a66c28b2d29ca75841b2c64a8b66873a0c56a8dcae8c67eff2bd14080e802e" Nov 01 23:09:39.947256 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.947227 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/129162f7-0fdd-426d-a66f-6acecdf14172-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/129162f7-0fdd-426d-a66f-6acecdf14172-maas-api-tls\") pod \"129162f7-0fdd-426d-a66f-6acecdf14172\" (UID: \"129162f7-0fdd-426d-a66f-6acecdf14172\") " Nov 01 23:09:39.947358 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.947324 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/129162f7-0fdd-426d-a66f-6acecdf14172-metrics-certs\" (UniqueName: \"kubernetes.io/secret/129162f7-0fdd-426d-a66f-6acecdf14172-metrics-certs\") pod \"129162f7-0fdd-426d-a66f-6acecdf14172\" (UID: \"129162f7-0fdd-426d-a66f-6acecdf14172\") " Nov 01 23:09:39.947504 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.947481 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/129162f7-0fdd-426d-a66f-6acecdf14172-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/129162f7-0fdd-426d-a66f-6acecdf14172-openshift-service-ca\") pod \"129162f7-0fdd-426d-a66f-6acecdf14172\" (UID: \"129162f7-0fdd-426d-a66f-6acecdf14172\") " Nov 01 23:09:39.947615 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.947598 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/129162f7-0fdd-426d-a66f-6acecdf14172-kube-api-access-vz7gp\" (UniqueName: \"kubernetes.io/projected/129162f7-0fdd-426d-a66f-6acecdf14172-kube-api-access-vz7gp\") pod \"129162f7-0fdd-426d-a66f-6acecdf14172\" (UID: \"129162f7-0fdd-426d-a66f-6acecdf14172\") " Nov 01 23:09:39.947923 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.947824 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/129162f7-0fdd-426d-a66f-6acecdf14172-openshift-service-ca" pod "129162f7-0fdd-426d-a66f-6acecdf14172" (UID: "129162f7-0fdd-426d-a66f-6acecdf14172"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:39.948399 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.948373 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/129162f7-0fdd-426d-a66f-6acecdf14172-openshift-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:39.949621 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.949588 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/129162f7-0fdd-426d-a66f-6acecdf14172-metrics-certs" pod "129162f7-0fdd-426d-a66f-6acecdf14172" (UID: "129162f7-0fdd-426d-a66f-6acecdf14172"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:09:39.950085 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.950063 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/129162f7-0fdd-426d-a66f-6acecdf14172-kube-api-access-vz7gp" pod "129162f7-0fdd-426d-a66f-6acecdf14172" (UID: "129162f7-0fdd-426d-a66f-6acecdf14172"). InnerVolumeSpecName "kube-api-access-vz7gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:39.950250 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.950231 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/129162f7-0fdd-426d-a66f-6acecdf14172-maas-api-tls" pod "129162f7-0fdd-426d-a66f-6acecdf14172" (UID: "129162f7-0fdd-426d-a66f-6acecdf14172"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:09:39.950586 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.950570 5011 scope.go:122] "RemoveContainer" containerID="e4a66c28b2d29ca75841b2c64a8b66873a0c56a8dcae8c67eff2bd14080e802e" Nov 01 23:09:39.950926 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:09:39.950900 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4a66c28b2d29ca75841b2c64a8b66873a0c56a8dcae8c67eff2bd14080e802e\": container with ID starting with e4a66c28b2d29ca75841b2c64a8b66873a0c56a8dcae8c67eff2bd14080e802e not found: ID does not exist" containerID="e4a66c28b2d29ca75841b2c64a8b66873a0c56a8dcae8c67eff2bd14080e802e" Nov 01 23:09:39.950984 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.950935 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4a66c28b2d29ca75841b2c64a8b66873a0c56a8dcae8c67eff2bd14080e802e"} err="failed to get container status \"e4a66c28b2d29ca75841b2c64a8b66873a0c56a8dcae8c67eff2bd14080e802e\": rpc error: code = NotFound desc = could not find container \"e4a66c28b2d29ca75841b2c64a8b66873a0c56a8dcae8c67eff2bd14080e802e\": container with ID starting with e4a66c28b2d29ca75841b2c64a8b66873a0c56a8dcae8c67eff2bd14080e802e not found: ID does not exist" Nov 01 23:09:39.952172 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.952151 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz"] Nov 01 23:09:39.960334 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:39.960308 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz"] Nov 01 23:09:40.001262 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:40.001228 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/premium-w6-927b68-kserve-7779999776-589fz" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 23:09:40.012898 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:40.012867 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz"] Nov 01 23:09:40.048969 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:40.028101 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-927b68/distinct-w6-927b68-kserve-85cd564dbc-w4dsz"] Nov 01 23:09:40.063917 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:40.063477 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/129162f7-0fdd-426d-a66f-6acecdf14172-maas-api-tls\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:40.063917 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:40.063516 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/129162f7-0fdd-426d-a66f-6acecdf14172-metrics-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:40.063917 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:40.063536 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz7gp\" (UniqueName: \"kubernetes.io/projected/129162f7-0fdd-426d-a66f-6acecdf14172-kube-api-access-vz7gp\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:40.258151 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:40.258121 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn"] Nov 01 23:09:40.262129 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:40.262105 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-788d7b-7bc5c59ccb-h6fzn"] Nov 01 23:09:41.416601 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:41.416567 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-788d7b-57874489c5-kkxpl" podUID="7e080376-e7a9-406c-b102-cc3cd7c5ebcb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:09:41.749021 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:41.748992 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="129162f7-0fdd-426d-a66f-6acecdf14172" path="/var/lib/kubelet/pods/129162f7-0fdd-426d-a66f-6acecdf14172/volumes" Nov 01 23:09:41.749510 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:41.749496 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" path="/var/lib/kubelet/pods/3508619c-ddff-4d19-8e29-a28cb562de3b/volumes" Nov 01 23:09:41.749941 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:41.749925 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" path="/var/lib/kubelet/pods/51df36c5-b9ce-4b2f-8525-5aa2d8508801/volumes" Nov 01 23:09:46.996998 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:09:46.996956 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:09:47.148461 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:47.148425 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj"] Nov 01 23:09:47.148705 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:47.148674 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" podUID="a03efd24-da7e-407d-896d-d22e46206274" containerName="istio-proxy" containerID="cri-o://96732250585ad7144915c56e1032f185f361916b09c50c87095fdddf59906e75" gracePeriod=30 Nov 01 23:09:47.914664 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:47.914622 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:09:47.914864 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:47.914689 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" podUID="a03efd24-da7e-407d-896d-d22e46206274" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:09:51.415815 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:51.415774 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-788d7b-57874489c5-kkxpl" podUID="7e080376-e7a9-406c-b102-cc3cd7c5ebcb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:09:52.452744 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:52.452717 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 23:09:52.583615 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:52.583578 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-workload-socket\") pod \"a03efd24-da7e-407d-896d-d22e46206274\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " Nov 01 23:09:52.583802 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:52.583623 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-istio-envoy\") pod \"a03efd24-da7e-407d-896d-d22e46206274\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " Nov 01 23:09:52.583802 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:52.583671 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-workload-certs\") pod \"a03efd24-da7e-407d-896d-d22e46206274\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " Nov 01 23:09:52.583802 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:52.583709 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a03efd24-da7e-407d-896d-d22e46206274-istio-token\" (UniqueName: \"kubernetes.io/projected/a03efd24-da7e-407d-896d-d22e46206274-istio-token\") pod \"a03efd24-da7e-407d-896d-d22e46206274\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " Nov 01 23:09:52.583802 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:52.583743 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a03efd24-da7e-407d-896d-d22e46206274-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03efd24-da7e-407d-896d-d22e46206274-istio-podinfo\") pod \"a03efd24-da7e-407d-896d-d22e46206274\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " Nov 01 23:09:52.584074 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:52.583899 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-credential-socket\") pod \"a03efd24-da7e-407d-896d-d22e46206274\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " Nov 01 23:09:52.584074 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:52.583925 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-workload-socket" pod "a03efd24-da7e-407d-896d-d22e46206274" (UID: "a03efd24-da7e-407d-896d-d22e46206274"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:52.584074 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:52.583963 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a03efd24-da7e-407d-896d-d22e46206274-kube-api-access-t5zch\" (UniqueName: \"kubernetes.io/projected/a03efd24-da7e-407d-896d-d22e46206274-kube-api-access-t5zch\") pod \"a03efd24-da7e-407d-896d-d22e46206274\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " Nov 01 23:09:52.584074 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:52.584040 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a03efd24-da7e-407d-896d-d22e46206274-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03efd24-da7e-407d-896d-d22e46206274-istiod-ca-cert\") pod \"a03efd24-da7e-407d-896d-d22e46206274\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " Nov 01 23:09:52.584316 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:52.584078 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-istio-data\") pod \"a03efd24-da7e-407d-896d-d22e46206274\" (UID: \"a03efd24-da7e-407d-896d-d22e46206274\") " Nov 01 23:09:52.584316 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:52.584125 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-workload-certs" pod "a03efd24-da7e-407d-896d-d22e46206274" (UID: "a03efd24-da7e-407d-896d-d22e46206274"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:52.584316 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:52.584187 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-credential-socket" pod "a03efd24-da7e-407d-896d-d22e46206274" (UID: "a03efd24-da7e-407d-896d-d22e46206274"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:52.584482 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:52.584425 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a03efd24-da7e-407d-896d-d22e46206274-istiod-ca-cert" pod "a03efd24-da7e-407d-896d-d22e46206274" (UID: "a03efd24-da7e-407d-896d-d22e46206274"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:52.584706 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:52.584490 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-istio-data" pod "a03efd24-da7e-407d-896d-d22e46206274" (UID: "a03efd24-da7e-407d-896d-d22e46206274"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:52.585046 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:52.585022 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:52.585146 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:52.585051 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:52.585146 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:52.585068 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03efd24-da7e-407d-896d-d22e46206274-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:52.585146 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:52.585084 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:52.585146 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:52.585097 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:52.586283 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:52.586259 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-istio-envoy" pod "a03efd24-da7e-407d-896d-d22e46206274" (UID: "a03efd24-da7e-407d-896d-d22e46206274"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:52.586392 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:52.586311 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a03efd24-da7e-407d-896d-d22e46206274-istio-podinfo" pod "a03efd24-da7e-407d-896d-d22e46206274" (UID: "a03efd24-da7e-407d-896d-d22e46206274"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:09:52.586672 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:52.586655 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a03efd24-da7e-407d-896d-d22e46206274-kube-api-access-t5zch" pod "a03efd24-da7e-407d-896d-d22e46206274" (UID: "a03efd24-da7e-407d-896d-d22e46206274"). InnerVolumeSpecName "kube-api-access-t5zch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:52.586747 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:52.586679 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a03efd24-da7e-407d-896d-d22e46206274-istio-token" pod "a03efd24-da7e-407d-896d-d22e46206274" (UID: "a03efd24-da7e-407d-896d-d22e46206274"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:52.686948 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:52.686860 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a03efd24-da7e-407d-896d-d22e46206274-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:52.686948 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:52.686891 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03efd24-da7e-407d-896d-d22e46206274-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:52.686948 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:52.686901 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5zch\" (UniqueName: \"kubernetes.io/projected/a03efd24-da7e-407d-896d-d22e46206274-kube-api-access-t5zch\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:52.686948 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:52.686911 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03efd24-da7e-407d-896d-d22e46206274-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:53.028512 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:53.028425 5011 generic.go:356] "Generic (PLEG): container finished" podID="a03efd24-da7e-407d-896d-d22e46206274" containerID="96732250585ad7144915c56e1032f185f361916b09c50c87095fdddf59906e75" exitCode=0 Nov 01 23:09:53.028512 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:53.028494 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" Nov 01 23:09:53.028512 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:53.028493 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" event={"ID":"a03efd24-da7e-407d-896d-d22e46206274","Type":"ContainerDied","Data":"96732250585ad7144915c56e1032f185f361916b09c50c87095fdddf59906e75"} Nov 01 23:09:53.028792 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:53.028558 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj" event={"ID":"a03efd24-da7e-407d-896d-d22e46206274","Type":"ContainerDied","Data":"04320d38c6c78a09141b44cbb4bdbe3243e5ae5d65f401d267c71936785d3589"} Nov 01 23:09:53.028792 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:53.028581 5011 scope.go:122] "RemoveContainer" containerID="96732250585ad7144915c56e1032f185f361916b09c50c87095fdddf59906e75" Nov 01 23:09:53.045393 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:53.045372 5011 scope.go:122] "RemoveContainer" containerID="53a9b720e37ce41acd3fb311faa168f34d9b506fa7322cbe682798a67b036677" Nov 01 23:09:53.056027 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:53.056004 5011 scope.go:122] "RemoveContainer" containerID="96732250585ad7144915c56e1032f185f361916b09c50c87095fdddf59906e75" Nov 01 23:09:53.056263 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:53.056239 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj"] Nov 01 23:09:53.056356 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:09:53.056332 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96732250585ad7144915c56e1032f185f361916b09c50c87095fdddf59906e75\": container with ID starting with 96732250585ad7144915c56e1032f185f361916b09c50c87095fdddf59906e75 not found: ID does not exist" containerID="96732250585ad7144915c56e1032f185f361916b09c50c87095fdddf59906e75" Nov 01 23:09:53.056405 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:53.056365 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96732250585ad7144915c56e1032f185f361916b09c50c87095fdddf59906e75"} err="failed to get container status \"96732250585ad7144915c56e1032f185f361916b09c50c87095fdddf59906e75\": rpc error: code = NotFound desc = could not find container \"96732250585ad7144915c56e1032f185f361916b09c50c87095fdddf59906e75\": container with ID starting with 96732250585ad7144915c56e1032f185f361916b09c50c87095fdddf59906e75 not found: ID does not exist" Nov 01 23:09:53.056405 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:53.056385 5011 scope.go:122] "RemoveContainer" containerID="53a9b720e37ce41acd3fb311faa168f34d9b506fa7322cbe682798a67b036677" Nov 01 23:09:53.057873 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:09:53.056791 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53a9b720e37ce41acd3fb311faa168f34d9b506fa7322cbe682798a67b036677\": container with ID starting with 53a9b720e37ce41acd3fb311faa168f34d9b506fa7322cbe682798a67b036677 not found: ID does not exist" containerID="53a9b720e37ce41acd3fb311faa168f34d9b506fa7322cbe682798a67b036677" Nov 01 23:09:53.057873 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:53.056849 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53a9b720e37ce41acd3fb311faa168f34d9b506fa7322cbe682798a67b036677"} err="failed to get container status \"53a9b720e37ce41acd3fb311faa168f34d9b506fa7322cbe682798a67b036677\": rpc error: code = NotFound desc = could not find container \"53a9b720e37ce41acd3fb311faa168f34d9b506fa7322cbe682798a67b036677\": container with ID starting with 53a9b720e37ce41acd3fb311faa168f34d9b506fa7322cbe682798a67b036677 not found: ID does not exist" Nov 01 23:09:53.060644 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:53.060625 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-gwdpj"] Nov 01 23:09:53.753926 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:53.753890 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a03efd24-da7e-407d-896d-d22e46206274" path="/var/lib/kubelet/pods/a03efd24-da7e-407d-896d-d22e46206274/volumes" Nov 01 23:09:56.944316 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:56.944279 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-788d7b-57874489c5-kkxpl" Nov 01 23:09:56.979927 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:56.979072 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e080376-e7a9-406c-b102-cc3cd7c5ebcb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e080376-e7a9-406c-b102-cc3cd7c5ebcb-plugins-config-volume\") pod \"7e080376-e7a9-406c-b102-cc3cd7c5ebcb\" (UID: \"7e080376-e7a9-406c-b102-cc3cd7c5ebcb\") " Nov 01 23:09:56.979927 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:56.979155 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e080376-e7a9-406c-b102-cc3cd7c5ebcb-kube-api-access-kd85d\" (UniqueName: \"kubernetes.io/projected/7e080376-e7a9-406c-b102-cc3cd7c5ebcb-kube-api-access-kd85d\") pod \"7e080376-e7a9-406c-b102-cc3cd7c5ebcb\" (UID: \"7e080376-e7a9-406c-b102-cc3cd7c5ebcb\") " Nov 01 23:09:56.979927 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:56.979449 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e080376-e7a9-406c-b102-cc3cd7c5ebcb-plugins-config-volume" pod "7e080376-e7a9-406c-b102-cc3cd7c5ebcb" (UID: "7e080376-e7a9-406c-b102-cc3cd7c5ebcb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:56.979927 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:56.979899 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e080376-e7a9-406c-b102-cc3cd7c5ebcb-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:56.981516 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:56.981486 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e080376-e7a9-406c-b102-cc3cd7c5ebcb-kube-api-access-kd85d" pod "7e080376-e7a9-406c-b102-cc3cd7c5ebcb" (UID: "7e080376-e7a9-406c-b102-cc3cd7c5ebcb"). InnerVolumeSpecName "kube-api-access-kd85d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:57.073911 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:57.073876 5011 generic.go:356] "Generic (PLEG): container finished" podID="7e080376-e7a9-406c-b102-cc3cd7c5ebcb" containerID="526310d78ebbd50c674182729be237cedfd15e6c93b8942204a226d28c19d9db" exitCode=0 Nov 01 23:09:57.074134 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:57.073918 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-788d7b-57874489c5-kkxpl" event={"ID":"7e080376-e7a9-406c-b102-cc3cd7c5ebcb","Type":"ContainerDied","Data":"526310d78ebbd50c674182729be237cedfd15e6c93b8942204a226d28c19d9db"} Nov 01 23:09:57.074134 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:57.073936 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-788d7b-57874489c5-kkxpl" Nov 01 23:09:57.074134 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:57.073984 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-788d7b-57874489c5-kkxpl" event={"ID":"7e080376-e7a9-406c-b102-cc3cd7c5ebcb","Type":"ContainerDied","Data":"b0f993cf54ea66c233b6e351c87fdc28ad25f74cb5884029a912671b919b67f1"} Nov 01 23:09:57.074134 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:57.074014 5011 scope.go:122] "RemoveContainer" containerID="526310d78ebbd50c674182729be237cedfd15e6c93b8942204a226d28c19d9db" Nov 01 23:09:57.081188 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:57.081157 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd85d\" (UniqueName: \"kubernetes.io/projected/7e080376-e7a9-406c-b102-cc3cd7c5ebcb-kube-api-access-kd85d\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:57.085598 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:57.085574 5011 scope.go:122] "RemoveContainer" containerID="526310d78ebbd50c674182729be237cedfd15e6c93b8942204a226d28c19d9db" Nov 01 23:09:57.085990 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:09:57.085969 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"526310d78ebbd50c674182729be237cedfd15e6c93b8942204a226d28c19d9db\": container with ID starting with 526310d78ebbd50c674182729be237cedfd15e6c93b8942204a226d28c19d9db not found: ID does not exist" containerID="526310d78ebbd50c674182729be237cedfd15e6c93b8942204a226d28c19d9db" Nov 01 23:09:57.086078 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:57.085998 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"526310d78ebbd50c674182729be237cedfd15e6c93b8942204a226d28c19d9db"} err="failed to get container status \"526310d78ebbd50c674182729be237cedfd15e6c93b8942204a226d28c19d9db\": rpc error: code = NotFound desc = could not find container \"526310d78ebbd50c674182729be237cedfd15e6c93b8942204a226d28c19d9db\": container with ID starting with 526310d78ebbd50c674182729be237cedfd15e6c93b8942204a226d28c19d9db not found: ID does not exist" Nov 01 23:09:57.110064 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:57.110031 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-788d7b-57874489c5-kkxpl"] Nov 01 23:09:57.117216 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:57.117187 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-788d7b-57874489c5-kkxpl"] Nov 01 23:09:57.748957 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:57.748926 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e080376-e7a9-406c-b102-cc3cd7c5ebcb" path="/var/lib/kubelet/pods/7e080376-e7a9-406c-b102-cc3cd7c5ebcb/volumes" Nov 01 23:09:57.749372 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:09:57.749352 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d73b25-7ffc64f5d5-7wcv8"] Nov 01 23:10:00.648158 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.648124 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-46ce02-6c8b5f89b8-wr5pn"] Nov 01 23:10:00.648813 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.648798 5011 cpu_manager.go:414] "containerMap: removing container" podUID="7e080376-e7a9-406c-b102-cc3cd7c5ebcb" containerName="payload-pre-processing" Nov 01 23:10:00.648916 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.648815 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e080376-e7a9-406c-b102-cc3cd7c5ebcb" containerName="payload-pre-processing" Nov 01 23:10:00.648916 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.648846 5011 cpu_manager.go:414] "containerMap: removing container" podUID="bc7cb565-6bf1-400f-9a3d-f062af8270bd" containerName="istio-proxy" Nov 01 23:10:00.648916 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.648852 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc7cb565-6bf1-400f-9a3d-f062af8270bd" containerName="istio-proxy" Nov 01 23:10:00.648916 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.648864 5011 cpu_manager.go:414] "containerMap: removing container" podUID="a32d183e-1080-4cef-ab97-f31aa667ef5c" containerName="istio-proxy" Nov 01 23:10:00.648916 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.648869 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a32d183e-1080-4cef-ab97-f31aa667ef5c" containerName="istio-proxy" Nov 01 23:10:00.648916 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.648879 5011 cpu_manager.go:414] "containerMap: removing container" podUID="a03efd24-da7e-407d-896d-d22e46206274" containerName="istio-proxy" Nov 01 23:10:00.648916 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.648884 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a03efd24-da7e-407d-896d-d22e46206274" containerName="istio-proxy" Nov 01 23:10:00.648916 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.648906 5011 cpu_manager.go:414] "containerMap: removing container" podUID="5dbf96e5-22cf-4e5b-8208-470f9c1f616c" containerName="payload-pre-processing" Nov 01 23:10:00.648916 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.648911 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dbf96e5-22cf-4e5b-8208-470f9c1f616c" containerName="payload-pre-processing" Nov 01 23:10:00.648916 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.648921 5011 cpu_manager.go:414] "containerMap: removing container" podUID="129162f7-0fdd-426d-a66f-6acecdf14172" containerName="maas-api" Nov 01 23:10:00.649248 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.648926 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="129162f7-0fdd-426d-a66f-6acecdf14172" containerName="maas-api" Nov 01 23:10:00.649248 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.648931 5011 cpu_manager.go:414] "containerMap: removing container" podUID="6f002a7d-fa04-46d4-a4ca-4df428b7e180" containerName="main" Nov 01 23:10:00.649248 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.648937 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f002a7d-fa04-46d4-a4ca-4df428b7e180" containerName="main" Nov 01 23:10:00.649248 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.648946 5011 cpu_manager.go:414] "containerMap: removing container" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" Nov 01 23:10:00.649248 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.648950 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" Nov 01 23:10:00.649248 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.648958 5011 cpu_manager.go:414] "containerMap: removing container" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" Nov 01 23:10:00.649248 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.648963 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" Nov 01 23:10:00.649248 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.648969 5011 cpu_manager.go:414] "containerMap: removing container" podUID="2c1fbfd9-c189-4b15-92aa-394d88d95ad8" containerName="payload-processing" Nov 01 23:10:00.649248 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.648974 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c1fbfd9-c189-4b15-92aa-394d88d95ad8" containerName="payload-processing" Nov 01 23:10:00.649248 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.649095 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="51df36c5-b9ce-4b2f-8525-5aa2d8508801" containerName="main" Nov 01 23:10:00.649248 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.649104 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="3508619c-ddff-4d19-8e29-a28cb562de3b" containerName="main" Nov 01 23:10:00.649248 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.649109 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c1fbfd9-c189-4b15-92aa-394d88d95ad8" containerName="payload-processing" Nov 01 23:10:00.649248 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.649118 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="a03efd24-da7e-407d-896d-d22e46206274" containerName="istio-proxy" Nov 01 23:10:00.649248 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.649124 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc7cb565-6bf1-400f-9a3d-f062af8270bd" containerName="istio-proxy" Nov 01 23:10:00.649248 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.649130 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e080376-e7a9-406c-b102-cc3cd7c5ebcb" containerName="payload-pre-processing" Nov 01 23:10:00.649248 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.649136 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc7cb565-6bf1-400f-9a3d-f062af8270bd" containerName="istio-proxy" Nov 01 23:10:00.649248 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.649141 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="a32d183e-1080-4cef-ab97-f31aa667ef5c" containerName="istio-proxy" Nov 01 23:10:00.649248 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.649149 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dbf96e5-22cf-4e5b-8208-470f9c1f616c" containerName="payload-pre-processing" Nov 01 23:10:00.649248 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.649156 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="129162f7-0fdd-426d-a66f-6acecdf14172" containerName="maas-api" Nov 01 23:10:00.649248 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.649161 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f002a7d-fa04-46d4-a4ca-4df428b7e180" containerName="main" Nov 01 23:10:00.654109 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.654088 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-6c8b5f89b8-wr5pn" Nov 01 23:10:00.665619 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.665318 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-46ce02-dockercfg-9czp8\"" Nov 01 23:10:00.677480 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.677447 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-46ce02-6c8b5f89b8-wr5pn"] Nov 01 23:10:00.771878 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.761126 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-46ce02\"" Nov 01 23:10:00.823544 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.822408 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptjbx\" (UniqueName: \"kubernetes.io/projected/e3a83ece-8434-49fc-a01f-2f926936384d-kube-api-access-ptjbx\") pod \"payload-processing-e2e-ipp-b-46ce02-6c8b5f89b8-wr5pn\" (UID: \"e3a83ece-8434-49fc-a01f-2f926936384d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-6c8b5f89b8-wr5pn" Nov 01 23:10:00.823544 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.822521 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3a83ece-8434-49fc-a01f-2f926936384d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-46ce02-6c8b5f89b8-wr5pn\" (UID: \"e3a83ece-8434-49fc-a01f-2f926936384d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-6c8b5f89b8-wr5pn" Nov 01 23:10:00.933501 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.931782 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptjbx\" (UniqueName: \"kubernetes.io/projected/e3a83ece-8434-49fc-a01f-2f926936384d-kube-api-access-ptjbx\") pod \"payload-processing-e2e-ipp-b-46ce02-6c8b5f89b8-wr5pn\" (UID: \"e3a83ece-8434-49fc-a01f-2f926936384d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-6c8b5f89b8-wr5pn" Nov 01 23:10:00.933888 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.933845 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3a83ece-8434-49fc-a01f-2f926936384d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-46ce02-6c8b5f89b8-wr5pn\" (UID: \"e3a83ece-8434-49fc-a01f-2f926936384d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-6c8b5f89b8-wr5pn" Nov 01 23:10:00.941897 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.935122 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3a83ece-8434-49fc-a01f-2f926936384d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-46ce02-6c8b5f89b8-wr5pn\" (UID: \"e3a83ece-8434-49fc-a01f-2f926936384d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-6c8b5f89b8-wr5pn" Nov 01 23:10:00.949524 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.949488 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptjbx\" (UniqueName: \"kubernetes.io/projected/e3a83ece-8434-49fc-a01f-2f926936384d-kube-api-access-ptjbx\") pod \"payload-processing-e2e-ipp-b-46ce02-6c8b5f89b8-wr5pn\" (UID: \"e3a83ece-8434-49fc-a01f-2f926936384d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-6c8b5f89b8-wr5pn" Nov 01 23:10:00.974108 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:00.974072 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-6c8b5f89b8-wr5pn" Nov 01 23:10:01.354904 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:01.353347 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-46ce02-6c8b5f89b8-wr5pn"] Nov 01 23:10:01.372858 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:10:01.358274 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a83ece_8434_49fc_a01f_2f926936384d.slice/crio-e6148c18e891a8d84444cafcde0c6b5ea69a2554195b184555b5d248ca4e460d WatchSource:0}: Error finding container e6148c18e891a8d84444cafcde0c6b5ea69a2554195b184555b5d248ca4e460d: Status 404 returned error can't find the container with id e6148c18e891a8d84444cafcde0c6b5ea69a2554195b184555b5d248ca4e460d Nov 01 23:10:02.133002 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:02.132969 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-6c8b5f89b8-wr5pn" event={"ID":"e3a83ece-8434-49fc-a01f-2f926936384d","Type":"ContainerStarted","Data":"e6148c18e891a8d84444cafcde0c6b5ea69a2554195b184555b5d248ca4e460d"} Nov 01 23:10:02.760013 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:02.758918 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d73b25-7ffc64f5d5-7wcv8" podUID="8fee5226-37e2-4c83-9225-449df015af93" containerName="payload-processing" containerID="cri-o://1f8067b1b20d768b1a845887a55a74547510e04d9a8b8de060b307782ea48728" gracePeriod=25 Nov 01 23:10:03.536391 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:03.536350 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj"] Nov 01 23:10:03.536928 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:03.536626 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" podUID="be85990a-36c7-4d36-8867-28b4c83867d8" containerName="istio-proxy" containerID="cri-o://bf9f10c16f9cdab3b85815869b4269f7d16cb3d55551c2a209f5befc81cb8d05" gracePeriod=30 Nov 01 23:10:04.300015 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:04.299975 5011 patch_prober.go:29] interesting pod/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:10:04.300233 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:04.300059 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" podUID="be85990a-36c7-4d36-8867-28b4c83867d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:10:04.617653 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:04.617568 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9"] Nov 01 23:10:04.618517 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:04.618448 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" podUID="0ee11042-8133-4e65-950e-224a5b4ca910" containerName="istio-proxy" containerID="cri-o://8b3baaf41808f3b42c8e358849ded2c8903da9824640aeaedcf5d9dfe5037a1f" gracePeriod=30 Nov 01 23:10:05.191227 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:05.191179 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-6c8b5f89b8-wr5pn" event={"ID":"e3a83ece-8434-49fc-a01f-2f926936384d","Type":"ContainerStarted","Data":"3daecde9e54598133e5820c8f7f4b431f8d973668e70a539cbfd8c39072bb218"} Nov 01 23:10:05.192374 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:05.192347 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-6c8b5f89b8-wr5pn" Nov 01 23:10:05.220517 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:05.220454 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-6c8b5f89b8-wr5pn" podStartSLOduration=1.8836563499999999 podStartE2EDuration="5.220433856s" totalImagesPullingTime="3.336777506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:10:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:10:05.215263001 +0000 UTC m=+2773.619853645" watchObservedRunningTime="2026-11-01 23:10:05.220433856 +0000 UTC m=+2773.625024490" Nov 01 23:10:08.773809 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:08.773770 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47"] Nov 01 23:10:08.774360 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:08.774092 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" podUID="d7b301bf-5694-40b2-8789-07555de1ed0f" containerName="istio-proxy" containerID="cri-o://5f3a28c4c719374f67c65e73ae62c9270d5d93664d4eef3f33b9914b7acb61e9" gracePeriod=30 Nov 01 23:10:09.054808 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.054776 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:10:09.075252 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.075222 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-workload-certs\") pod \"be85990a-36c7-4d36-8867-28b4c83867d8\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " Nov 01 23:10:09.075473 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.075285 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-credential-socket\") pod \"be85990a-36c7-4d36-8867-28b4c83867d8\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " Nov 01 23:10:09.075473 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.075381 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be85990a-36c7-4d36-8867-28b4c83867d8-kube-api-access-9x4dh\" (UniqueName: \"kubernetes.io/projected/be85990a-36c7-4d36-8867-28b4c83867d8-kube-api-access-9x4dh\") pod \"be85990a-36c7-4d36-8867-28b4c83867d8\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " Nov 01 23:10:09.075473 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.075398 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-workload-certs" pod "be85990a-36c7-4d36-8867-28b4c83867d8" (UID: "be85990a-36c7-4d36-8867-28b4c83867d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:09.075473 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.075410 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be85990a-36c7-4d36-8867-28b4c83867d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be85990a-36c7-4d36-8867-28b4c83867d8-istiod-ca-cert\") pod \"be85990a-36c7-4d36-8867-28b4c83867d8\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " Nov 01 23:10:09.075473 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.075464 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be85990a-36c7-4d36-8867-28b4c83867d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be85990a-36c7-4d36-8867-28b4c83867d8-istio-podinfo\") pod \"be85990a-36c7-4d36-8867-28b4c83867d8\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " Nov 01 23:10:09.075750 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.075562 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-credential-socket" pod "be85990a-36c7-4d36-8867-28b4c83867d8" (UID: "be85990a-36c7-4d36-8867-28b4c83867d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:09.075750 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.075636 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-istio-envoy\") pod \"be85990a-36c7-4d36-8867-28b4c83867d8\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " Nov 01 23:10:09.075750 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.075726 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-istio-data\") pod \"be85990a-36c7-4d36-8867-28b4c83867d8\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " Nov 01 23:10:09.075750 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.075736 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be85990a-36c7-4d36-8867-28b4c83867d8-istiod-ca-cert" pod "be85990a-36c7-4d36-8867-28b4c83867d8" (UID: "be85990a-36c7-4d36-8867-28b4c83867d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:09.075957 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.075788 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-workload-socket\") pod \"be85990a-36c7-4d36-8867-28b4c83867d8\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " Nov 01 23:10:09.075957 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.075921 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be85990a-36c7-4d36-8867-28b4c83867d8-istio-token\" (UniqueName: \"kubernetes.io/projected/be85990a-36c7-4d36-8867-28b4c83867d8-istio-token\") pod \"be85990a-36c7-4d36-8867-28b4c83867d8\" (UID: \"be85990a-36c7-4d36-8867-28b4c83867d8\") " Nov 01 23:10:09.076090 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.076069 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-istio-data" pod "be85990a-36c7-4d36-8867-28b4c83867d8" (UID: "be85990a-36c7-4d36-8867-28b4c83867d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:09.076516 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.076380 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-workload-socket" pod "be85990a-36c7-4d36-8867-28b4c83867d8" (UID: "be85990a-36c7-4d36-8867-28b4c83867d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:09.076516 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.076469 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:09.076516 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.076485 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:09.076516 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.076498 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:09.076516 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.076506 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be85990a-36c7-4d36-8867-28b4c83867d8-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:09.078286 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.078191 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-istio-envoy" pod "be85990a-36c7-4d36-8867-28b4c83867d8" (UID: "be85990a-36c7-4d36-8867-28b4c83867d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:09.078286 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.078197 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be85990a-36c7-4d36-8867-28b4c83867d8-kube-api-access-9x4dh" pod "be85990a-36c7-4d36-8867-28b4c83867d8" (UID: "be85990a-36c7-4d36-8867-28b4c83867d8"). InnerVolumeSpecName "kube-api-access-9x4dh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:09.078286 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.078251 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be85990a-36c7-4d36-8867-28b4c83867d8-istio-podinfo" pod "be85990a-36c7-4d36-8867-28b4c83867d8" (UID: "be85990a-36c7-4d36-8867-28b4c83867d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:10:09.078625 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.078605 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be85990a-36c7-4d36-8867-28b4c83867d8-istio-token" pod "be85990a-36c7-4d36-8867-28b4c83867d8" (UID: "be85990a-36c7-4d36-8867-28b4c83867d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:09.178726 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.178683 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be85990a-36c7-4d36-8867-28b4c83867d8-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:09.178726 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.178727 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x4dh\" (UniqueName: \"kubernetes.io/projected/be85990a-36c7-4d36-8867-28b4c83867d8-kube-api-access-9x4dh\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:09.178982 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.178745 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be85990a-36c7-4d36-8867-28b4c83867d8-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:09.178982 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.178761 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:09.178982 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.178778 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be85990a-36c7-4d36-8867-28b4c83867d8-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:09.264161 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.264061 5011 generic.go:356] "Generic (PLEG): container finished" podID="be85990a-36c7-4d36-8867-28b4c83867d8" containerID="bf9f10c16f9cdab3b85815869b4269f7d16cb3d55551c2a209f5befc81cb8d05" exitCode=0 Nov 01 23:10:09.264161 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.264114 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" Nov 01 23:10:09.264161 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.264148 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" event={"ID":"be85990a-36c7-4d36-8867-28b4c83867d8","Type":"ContainerDied","Data":"bf9f10c16f9cdab3b85815869b4269f7d16cb3d55551c2a209f5befc81cb8d05"} Nov 01 23:10:09.264470 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.264213 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj" event={"ID":"be85990a-36c7-4d36-8867-28b4c83867d8","Type":"ContainerDied","Data":"9ae83a96a2a665e2d73c5b1fd6203891d5047ac4dac29cd0a6a5de745dc0a4a7"} Nov 01 23:10:09.264470 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.264248 5011 scope.go:122] "RemoveContainer" containerID="bf9f10c16f9cdab3b85815869b4269f7d16cb3d55551c2a209f5befc81cb8d05" Nov 01 23:10:09.276440 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.276412 5011 scope.go:122] "RemoveContainer" containerID="92340491393b1c93d8968984727b54f3cd6f2e591097756d6138434964ae3988" Nov 01 23:10:09.290431 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.290400 5011 scope.go:122] "RemoveContainer" containerID="bf9f10c16f9cdab3b85815869b4269f7d16cb3d55551c2a209f5befc81cb8d05" Nov 01 23:10:09.290898 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:10:09.290825 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf9f10c16f9cdab3b85815869b4269f7d16cb3d55551c2a209f5befc81cb8d05\": container with ID starting with bf9f10c16f9cdab3b85815869b4269f7d16cb3d55551c2a209f5befc81cb8d05 not found: ID does not exist" containerID="bf9f10c16f9cdab3b85815869b4269f7d16cb3d55551c2a209f5befc81cb8d05" Nov 01 23:10:09.291017 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.290913 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf9f10c16f9cdab3b85815869b4269f7d16cb3d55551c2a209f5befc81cb8d05"} err="failed to get container status \"bf9f10c16f9cdab3b85815869b4269f7d16cb3d55551c2a209f5befc81cb8d05\": rpc error: code = NotFound desc = could not find container \"bf9f10c16f9cdab3b85815869b4269f7d16cb3d55551c2a209f5befc81cb8d05\": container with ID starting with bf9f10c16f9cdab3b85815869b4269f7d16cb3d55551c2a209f5befc81cb8d05 not found: ID does not exist" Nov 01 23:10:09.291017 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.290942 5011 scope.go:122] "RemoveContainer" containerID="92340491393b1c93d8968984727b54f3cd6f2e591097756d6138434964ae3988" Nov 01 23:10:09.291278 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:10:09.291251 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92340491393b1c93d8968984727b54f3cd6f2e591097756d6138434964ae3988\": container with ID starting with 92340491393b1c93d8968984727b54f3cd6f2e591097756d6138434964ae3988 not found: ID does not exist" containerID="92340491393b1c93d8968984727b54f3cd6f2e591097756d6138434964ae3988" Nov 01 23:10:09.291355 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.291290 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92340491393b1c93d8968984727b54f3cd6f2e591097756d6138434964ae3988"} err="failed to get container status \"92340491393b1c93d8968984727b54f3cd6f2e591097756d6138434964ae3988\": rpc error: code = NotFound desc = could not find container \"92340491393b1c93d8968984727b54f3cd6f2e591097756d6138434964ae3988\": container with ID starting with 92340491393b1c93d8968984727b54f3cd6f2e591097756d6138434964ae3988 not found: ID does not exist" Nov 01 23:10:09.297557 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.297533 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj"] Nov 01 23:10:09.306032 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.306009 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-337dee-openshift-default-69b4b7c8cc-wr5hj"] Nov 01 23:10:09.750316 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:09.750258 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be85990a-36c7-4d36-8867-28b4c83867d8" path="/var/lib/kubelet/pods/be85990a-36c7-4d36-8867-28b4c83867d8/volumes" Nov 01 23:10:10.015625 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.015591 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 23:10:10.093381 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.093338 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-workload-socket\") pod \"0ee11042-8133-4e65-950e-224a5b4ca910\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " Nov 01 23:10:10.093611 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.093532 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ee11042-8133-4e65-950e-224a5b4ca910-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ee11042-8133-4e65-950e-224a5b4ca910-istio-podinfo\") pod \"0ee11042-8133-4e65-950e-224a5b4ca910\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " Nov 01 23:10:10.093705 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.093608 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-istio-envoy\") pod \"0ee11042-8133-4e65-950e-224a5b4ca910\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " Nov 01 23:10:10.093705 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.093660 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-workload-socket" pod "0ee11042-8133-4e65-950e-224a5b4ca910" (UID: "0ee11042-8133-4e65-950e-224a5b4ca910"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:10.093875 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.093735 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ee11042-8133-4e65-950e-224a5b4ca910-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ee11042-8133-4e65-950e-224a5b4ca910-istiod-ca-cert\") pod \"0ee11042-8133-4e65-950e-224a5b4ca910\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " Nov 01 23:10:10.093875 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.093770 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-istio-data\") pod \"0ee11042-8133-4e65-950e-224a5b4ca910\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " Nov 01 23:10:10.094000 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.093895 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ee11042-8133-4e65-950e-224a5b4ca910-istio-token\" (UniqueName: \"kubernetes.io/projected/0ee11042-8133-4e65-950e-224a5b4ca910-istio-token\") pod \"0ee11042-8133-4e65-950e-224a5b4ca910\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " Nov 01 23:10:10.094000 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.093951 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-workload-certs\") pod \"0ee11042-8133-4e65-950e-224a5b4ca910\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " Nov 01 23:10:10.094110 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.094000 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ee11042-8133-4e65-950e-224a5b4ca910-kube-api-access-dzxcr\" (UniqueName: \"kubernetes.io/projected/0ee11042-8133-4e65-950e-224a5b4ca910-kube-api-access-dzxcr\") pod \"0ee11042-8133-4e65-950e-224a5b4ca910\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " Nov 01 23:10:10.094110 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.094033 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-credential-socket\") pod \"0ee11042-8133-4e65-950e-224a5b4ca910\" (UID: \"0ee11042-8133-4e65-950e-224a5b4ca910\") " Nov 01 23:10:10.094572 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.094336 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ee11042-8133-4e65-950e-224a5b4ca910-istiod-ca-cert" pod "0ee11042-8133-4e65-950e-224a5b4ca910" (UID: "0ee11042-8133-4e65-950e-224a5b4ca910"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:10.094808 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.094784 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-credential-socket" pod "0ee11042-8133-4e65-950e-224a5b4ca910" (UID: "0ee11042-8133-4e65-950e-224a5b4ca910"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:10.095088 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.094992 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-istio-data" pod "0ee11042-8133-4e65-950e-224a5b4ca910" (UID: "0ee11042-8133-4e65-950e-224a5b4ca910"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:10.095088 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.095037 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:10.095088 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.095077 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ee11042-8133-4e65-950e-224a5b4ca910-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:10.096115 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.096086 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-workload-certs" pod "0ee11042-8133-4e65-950e-224a5b4ca910" (UID: "0ee11042-8133-4e65-950e-224a5b4ca910"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:10.097091 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.096932 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ee11042-8133-4e65-950e-224a5b4ca910-istio-podinfo" pod "0ee11042-8133-4e65-950e-224a5b4ca910" (UID: "0ee11042-8133-4e65-950e-224a5b4ca910"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:10:10.097665 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.097645 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ee11042-8133-4e65-950e-224a5b4ca910-istio-token" pod "0ee11042-8133-4e65-950e-224a5b4ca910" (UID: "0ee11042-8133-4e65-950e-224a5b4ca910"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:10.097760 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.097702 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-istio-envoy" pod "0ee11042-8133-4e65-950e-224a5b4ca910" (UID: "0ee11042-8133-4e65-950e-224a5b4ca910"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:10.098055 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.098031 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ee11042-8133-4e65-950e-224a5b4ca910-kube-api-access-dzxcr" pod "0ee11042-8133-4e65-950e-224a5b4ca910" (UID: "0ee11042-8133-4e65-950e-224a5b4ca910"). InnerVolumeSpecName "kube-api-access-dzxcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:10.197058 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.197021 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ee11042-8133-4e65-950e-224a5b4ca910-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:10.197058 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.197052 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:10.197058 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.197060 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:10.197306 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.197069 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ee11042-8133-4e65-950e-224a5b4ca910-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:10.197306 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.197078 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:10.197306 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.197087 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzxcr\" (UniqueName: \"kubernetes.io/projected/0ee11042-8133-4e65-950e-224a5b4ca910-kube-api-access-dzxcr\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:10.197306 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.197096 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee11042-8133-4e65-950e-224a5b4ca910-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:10.276404 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.276317 5011 generic.go:356] "Generic (PLEG): container finished" podID="0ee11042-8133-4e65-950e-224a5b4ca910" containerID="8b3baaf41808f3b42c8e358849ded2c8903da9824640aeaedcf5d9dfe5037a1f" exitCode=0 Nov 01 23:10:10.276404 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.276387 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" Nov 01 23:10:10.276633 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.276399 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" event={"ID":"0ee11042-8133-4e65-950e-224a5b4ca910","Type":"ContainerDied","Data":"8b3baaf41808f3b42c8e358849ded2c8903da9824640aeaedcf5d9dfe5037a1f"} Nov 01 23:10:10.276633 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.276481 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9" event={"ID":"0ee11042-8133-4e65-950e-224a5b4ca910","Type":"ContainerDied","Data":"367f6400b09671c3aaf50220bdd93a100d2b64d2a0de848ff6d05d48a30f9d51"} Nov 01 23:10:10.276633 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.276514 5011 scope.go:122] "RemoveContainer" containerID="8b3baaf41808f3b42c8e358849ded2c8903da9824640aeaedcf5d9dfe5037a1f" Nov 01 23:10:10.291263 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.291240 5011 scope.go:122] "RemoveContainer" containerID="8b3baaf41808f3b42c8e358849ded2c8903da9824640aeaedcf5d9dfe5037a1f" Nov 01 23:10:10.291597 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:10:10.291579 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b3baaf41808f3b42c8e358849ded2c8903da9824640aeaedcf5d9dfe5037a1f\": container with ID starting with 8b3baaf41808f3b42c8e358849ded2c8903da9824640aeaedcf5d9dfe5037a1f not found: ID does not exist" containerID="8b3baaf41808f3b42c8e358849ded2c8903da9824640aeaedcf5d9dfe5037a1f" Nov 01 23:10:10.291662 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.291607 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b3baaf41808f3b42c8e358849ded2c8903da9824640aeaedcf5d9dfe5037a1f"} err="failed to get container status \"8b3baaf41808f3b42c8e358849ded2c8903da9824640aeaedcf5d9dfe5037a1f\": rpc error: code = NotFound desc = could not find container \"8b3baaf41808f3b42c8e358849ded2c8903da9824640aeaedcf5d9dfe5037a1f\": container with ID starting with 8b3baaf41808f3b42c8e358849ded2c8903da9824640aeaedcf5d9dfe5037a1f not found: ID does not exist" Nov 01 23:10:10.306095 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.306065 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9"] Nov 01 23:10:10.310665 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:10.310636 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-p4px9"] Nov 01 23:10:11.755021 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:11.754998 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ee11042-8133-4e65-950e-224a5b4ca910" path="/var/lib/kubelet/pods/0ee11042-8133-4e65-950e-224a5b4ca910/volumes" Nov 01 23:10:12.077552 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:12.077454 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d73b25-7ffc64f5d5-7wcv8" podUID="8fee5226-37e2-4c83-9225-449df015af93" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:10:12.298921 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:12.298884 5011 generic.go:356] "Generic (PLEG): container finished" podID="d295266d-5d50-4aae-a0ee-d86513a36f34" containerID="88d917aa5db9d3054d9536a28d70c0834b7e5c53af461700d1c6f3fcce812380" exitCode=0 Nov 01 23:10:12.299097 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:12.298958 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" event={"ID":"d295266d-5d50-4aae-a0ee-d86513a36f34","Type":"ContainerDied","Data":"88d917aa5db9d3054d9536a28d70c0834b7e5c53af461700d1c6f3fcce812380"} Nov 01 23:10:12.299470 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:12.299455 5011 scope.go:122] "RemoveContainer" containerID="88d917aa5db9d3054d9536a28d70c0834b7e5c53af461700d1c6f3fcce812380" Nov 01 23:10:12.301311 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:12.301281 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:10:12.301393 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:12.301352 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:10:12.301393 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:12.301384 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:10:13.308379 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:13.308345 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" event={"ID":"d295266d-5d50-4aae-a0ee-d86513a36f34","Type":"ContainerStarted","Data":"f545ecb3a6ae9c637aa4de7689c261e1f5b9833cd6d45d0a6ebf21c12ad15b8d"} Nov 01 23:10:13.681067 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:13.681037 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:10:13.682141 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:13.682114 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 01 23:10:13.682284 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:13.682169 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" podUID="d295266d-5d50-4aae-a0ee-d86513a36f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 01 23:10:14.111722 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.111696 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 23:10:14.186801 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.186757 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-istio-envoy\") pod \"d7b301bf-5694-40b2-8789-07555de1ed0f\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " Nov 01 23:10:14.187037 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.186919 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-istio-data\") pod \"d7b301bf-5694-40b2-8789-07555de1ed0f\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " Nov 01 23:10:14.187037 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.186958 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7b301bf-5694-40b2-8789-07555de1ed0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b301bf-5694-40b2-8789-07555de1ed0f-istio-podinfo\") pod \"d7b301bf-5694-40b2-8789-07555de1ed0f\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " Nov 01 23:10:14.187037 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.187013 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7b301bf-5694-40b2-8789-07555de1ed0f-istio-token\" (UniqueName: \"kubernetes.io/projected/d7b301bf-5694-40b2-8789-07555de1ed0f-istio-token\") pod \"d7b301bf-5694-40b2-8789-07555de1ed0f\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " Nov 01 23:10:14.187235 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.187071 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-credential-socket\") pod \"d7b301bf-5694-40b2-8789-07555de1ed0f\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " Nov 01 23:10:14.187235 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.187163 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-workload-socket\") pod \"d7b301bf-5694-40b2-8789-07555de1ed0f\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " Nov 01 23:10:14.187235 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.187186 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7b301bf-5694-40b2-8789-07555de1ed0f-kube-api-access-h547g\" (UniqueName: \"kubernetes.io/projected/d7b301bf-5694-40b2-8789-07555de1ed0f-kube-api-access-h547g\") pod \"d7b301bf-5694-40b2-8789-07555de1ed0f\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " Nov 01 23:10:14.187400 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.187237 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-workload-certs\") pod \"d7b301bf-5694-40b2-8789-07555de1ed0f\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " Nov 01 23:10:14.187400 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.187231 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-istio-data" pod "d7b301bf-5694-40b2-8789-07555de1ed0f" (UID: "d7b301bf-5694-40b2-8789-07555de1ed0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:14.187400 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.187263 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7b301bf-5694-40b2-8789-07555de1ed0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b301bf-5694-40b2-8789-07555de1ed0f-istiod-ca-cert\") pod \"d7b301bf-5694-40b2-8789-07555de1ed0f\" (UID: \"d7b301bf-5694-40b2-8789-07555de1ed0f\") " Nov 01 23:10:14.187594 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.187450 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-workload-socket" pod "d7b301bf-5694-40b2-8789-07555de1ed0f" (UID: "d7b301bf-5694-40b2-8789-07555de1ed0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:14.187804 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.187770 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-workload-certs" pod "d7b301bf-5694-40b2-8789-07555de1ed0f" (UID: "d7b301bf-5694-40b2-8789-07555de1ed0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:14.188041 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.188014 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-credential-socket" pod "d7b301bf-5694-40b2-8789-07555de1ed0f" (UID: "d7b301bf-5694-40b2-8789-07555de1ed0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:14.188129 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.188037 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:14.188129 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.188046 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7b301bf-5694-40b2-8789-07555de1ed0f-istiod-ca-cert" pod "d7b301bf-5694-40b2-8789-07555de1ed0f" (UID: "d7b301bf-5694-40b2-8789-07555de1ed0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:14.188129 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.188060 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:14.188129 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.188077 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:14.189914 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.189881 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-istio-envoy" pod "d7b301bf-5694-40b2-8789-07555de1ed0f" (UID: "d7b301bf-5694-40b2-8789-07555de1ed0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:14.190211 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.190182 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b301bf-5694-40b2-8789-07555de1ed0f-istio-token" pod "d7b301bf-5694-40b2-8789-07555de1ed0f" (UID: "d7b301bf-5694-40b2-8789-07555de1ed0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:14.190314 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.190263 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7b301bf-5694-40b2-8789-07555de1ed0f-istio-podinfo" pod "d7b301bf-5694-40b2-8789-07555de1ed0f" (UID: "d7b301bf-5694-40b2-8789-07555de1ed0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:10:14.190314 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.190299 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b301bf-5694-40b2-8789-07555de1ed0f-kube-api-access-h547g" pod "d7b301bf-5694-40b2-8789-07555de1ed0f" (UID: "d7b301bf-5694-40b2-8789-07555de1ed0f"). InnerVolumeSpecName "kube-api-access-h547g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:14.225191 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.225096 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn"] Nov 01 23:10:14.267854 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.263809 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t"] Nov 01 23:10:14.289247 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.289215 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h547g\" (UniqueName: \"kubernetes.io/projected/d7b301bf-5694-40b2-8789-07555de1ed0f-kube-api-access-h547g\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:14.289247 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.289244 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7b301bf-5694-40b2-8789-07555de1ed0f-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:14.289247 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.289254 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:14.289471 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.289264 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7b301bf-5694-40b2-8789-07555de1ed0f-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:14.289471 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.289272 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7b301bf-5694-40b2-8789-07555de1ed0f-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:14.289471 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.289281 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7b301bf-5694-40b2-8789-07555de1ed0f-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:14.316344 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.316312 5011 generic.go:356] "Generic (PLEG): container finished" podID="d7b301bf-5694-40b2-8789-07555de1ed0f" containerID="5f3a28c4c719374f67c65e73ae62c9270d5d93664d4eef3f33b9914b7acb61e9" exitCode=0 Nov 01 23:10:14.316727 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.316359 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" Nov 01 23:10:14.316727 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.316398 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" event={"ID":"d7b301bf-5694-40b2-8789-07555de1ed0f","Type":"ContainerDied","Data":"5f3a28c4c719374f67c65e73ae62c9270d5d93664d4eef3f33b9914b7acb61e9"} Nov 01 23:10:14.316727 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.316457 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47" event={"ID":"d7b301bf-5694-40b2-8789-07555de1ed0f","Type":"ContainerDied","Data":"ad1b910f015be24b01f78d0adf542821c09005b7ca4308df0470952b83b08e2f"} Nov 01 23:10:14.316727 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.316486 5011 scope.go:122] "RemoveContainer" containerID="5f3a28c4c719374f67c65e73ae62c9270d5d93664d4eef3f33b9914b7acb61e9" Nov 01 23:10:14.328965 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.328822 5011 scope.go:122] "RemoveContainer" containerID="10375264c2c86cd17cb06109c5d4e5876ab93023af79de115915f4008c874104" Nov 01 23:10:14.352877 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.346367 5011 scope.go:122] "RemoveContainer" containerID="5f3a28c4c719374f67c65e73ae62c9270d5d93664d4eef3f33b9914b7acb61e9" Nov 01 23:10:14.354261 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:10:14.353948 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f3a28c4c719374f67c65e73ae62c9270d5d93664d4eef3f33b9914b7acb61e9\": container with ID starting with 5f3a28c4c719374f67c65e73ae62c9270d5d93664d4eef3f33b9914b7acb61e9 not found: ID does not exist" containerID="5f3a28c4c719374f67c65e73ae62c9270d5d93664d4eef3f33b9914b7acb61e9" Nov 01 23:10:14.354261 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.353989 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f3a28c4c719374f67c65e73ae62c9270d5d93664d4eef3f33b9914b7acb61e9"} err="failed to get container status \"5f3a28c4c719374f67c65e73ae62c9270d5d93664d4eef3f33b9914b7acb61e9\": rpc error: code = NotFound desc = could not find container \"5f3a28c4c719374f67c65e73ae62c9270d5d93664d4eef3f33b9914b7acb61e9\": container with ID starting with 5f3a28c4c719374f67c65e73ae62c9270d5d93664d4eef3f33b9914b7acb61e9 not found: ID does not exist" Nov 01 23:10:14.354261 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.354014 5011 scope.go:122] "RemoveContainer" containerID="10375264c2c86cd17cb06109c5d4e5876ab93023af79de115915f4008c874104" Nov 01 23:10:14.354694 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:10:14.354607 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10375264c2c86cd17cb06109c5d4e5876ab93023af79de115915f4008c874104\": container with ID starting with 10375264c2c86cd17cb06109c5d4e5876ab93023af79de115915f4008c874104 not found: ID does not exist" containerID="10375264c2c86cd17cb06109c5d4e5876ab93023af79de115915f4008c874104" Nov 01 23:10:14.354694 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.354639 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10375264c2c86cd17cb06109c5d4e5876ab93023af79de115915f4008c874104"} err="failed to get container status \"10375264c2c86cd17cb06109c5d4e5876ab93023af79de115915f4008c874104\": rpc error: code = NotFound desc = could not find container \"10375264c2c86cd17cb06109c5d4e5876ab93023af79de115915f4008c874104\": container with ID starting with 10375264c2c86cd17cb06109c5d4e5876ab93023af79de115915f4008c874104 not found: ID does not exist" Nov 01 23:10:14.372210 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.372177 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47"] Nov 01 23:10:14.376310 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.376279 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-9rf47"] Nov 01 23:10:14.390607 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.390535 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:10:14.681206 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.681176 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 01 23:10:14.681421 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:14.681235 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" podUID="d295266d-5d50-4aae-a0ee-d86513a36f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 01 23:10:15.681434 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:15.681403 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 01 23:10:15.681924 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:15.681459 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" podUID="d295266d-5d50-4aae-a0ee-d86513a36f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 01 23:10:15.771587 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:15.771555 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7b301bf-5694-40b2-8789-07555de1ed0f" path="/var/lib/kubelet/pods/d7b301bf-5694-40b2-8789-07555de1ed0f/volumes" Nov 01 23:10:16.681061 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:16.681029 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 01 23:10:16.681245 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:16.681091 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" podUID="d295266d-5d50-4aae-a0ee-d86513a36f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 01 23:10:17.012965 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:10:17.012021 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:10:17.211232 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:17.211203 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-6c8b5f89b8-wr5pn" Nov 01 23:10:17.681323 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:17.681291 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 01 23:10:17.681511 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:17.681344 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" podUID="d295266d-5d50-4aae-a0ee-d86513a36f34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 01 23:10:18.685635 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:18.685607 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:10:18.686916 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:18.686898 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:10:22.078164 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:22.078128 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d73b25-7ffc64f5d5-7wcv8" podUID="8fee5226-37e2-4c83-9225-449df015af93" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:10:23.063364 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:23.063335 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d73b25-7ffc64f5d5-7wcv8" Nov 01 23:10:23.157326 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:23.157296 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fee5226-37e2-4c83-9225-449df015af93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fee5226-37e2-4c83-9225-449df015af93-plugins-config-volume\") pod \"8fee5226-37e2-4c83-9225-449df015af93\" (UID: \"8fee5226-37e2-4c83-9225-449df015af93\") " Nov 01 23:10:23.157756 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:23.157348 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fee5226-37e2-4c83-9225-449df015af93-kube-api-access-79ppt\" (UniqueName: \"kubernetes.io/projected/8fee5226-37e2-4c83-9225-449df015af93-kube-api-access-79ppt\") pod \"8fee5226-37e2-4c83-9225-449df015af93\" (UID: \"8fee5226-37e2-4c83-9225-449df015af93\") " Nov 01 23:10:23.157756 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:23.157668 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fee5226-37e2-4c83-9225-449df015af93-plugins-config-volume" pod "8fee5226-37e2-4c83-9225-449df015af93" (UID: "8fee5226-37e2-4c83-9225-449df015af93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:23.158158 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:23.158138 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fee5226-37e2-4c83-9225-449df015af93-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:23.159818 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:23.159798 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fee5226-37e2-4c83-9225-449df015af93-kube-api-access-79ppt" pod "8fee5226-37e2-4c83-9225-449df015af93" (UID: "8fee5226-37e2-4c83-9225-449df015af93"). InnerVolumeSpecName "kube-api-access-79ppt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:23.259717 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:23.259685 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79ppt\" (UniqueName: \"kubernetes.io/projected/8fee5226-37e2-4c83-9225-449df015af93-kube-api-access-79ppt\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:23.390241 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:23.390140 5011 generic.go:356] "Generic (PLEG): container finished" podID="8fee5226-37e2-4c83-9225-449df015af93" containerID="1f8067b1b20d768b1a845887a55a74547510e04d9a8b8de060b307782ea48728" exitCode=0 Nov 01 23:10:23.390241 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:23.390186 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d73b25-7ffc64f5d5-7wcv8" Nov 01 23:10:23.390241 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:23.390230 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d73b25-7ffc64f5d5-7wcv8" event={"ID":"8fee5226-37e2-4c83-9225-449df015af93","Type":"ContainerDied","Data":"1f8067b1b20d768b1a845887a55a74547510e04d9a8b8de060b307782ea48728"} Nov 01 23:10:23.390482 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:23.390288 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d73b25-7ffc64f5d5-7wcv8" event={"ID":"8fee5226-37e2-4c83-9225-449df015af93","Type":"ContainerDied","Data":"a14294edfffc41182f8ee6740a4ef7c32f8bd806fe45d7f72e94f5c8af183c4e"} Nov 01 23:10:23.390482 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:23.390315 5011 scope.go:122] "RemoveContainer" containerID="1f8067b1b20d768b1a845887a55a74547510e04d9a8b8de060b307782ea48728" Nov 01 23:10:23.401631 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:23.401612 5011 scope.go:122] "RemoveContainer" containerID="1f8067b1b20d768b1a845887a55a74547510e04d9a8b8de060b307782ea48728" Nov 01 23:10:23.401957 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:10:23.401937 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f8067b1b20d768b1a845887a55a74547510e04d9a8b8de060b307782ea48728\": container with ID starting with 1f8067b1b20d768b1a845887a55a74547510e04d9a8b8de060b307782ea48728 not found: ID does not exist" containerID="1f8067b1b20d768b1a845887a55a74547510e04d9a8b8de060b307782ea48728" Nov 01 23:10:23.402026 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:23.401966 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f8067b1b20d768b1a845887a55a74547510e04d9a8b8de060b307782ea48728"} err="failed to get container status \"1f8067b1b20d768b1a845887a55a74547510e04d9a8b8de060b307782ea48728\": rpc error: code = NotFound desc = could not find container \"1f8067b1b20d768b1a845887a55a74547510e04d9a8b8de060b307782ea48728\": container with ID starting with 1f8067b1b20d768b1a845887a55a74547510e04d9a8b8de060b307782ea48728 not found: ID does not exist" Nov 01 23:10:23.413876 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:23.413851 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d73b25-7ffc64f5d5-7wcv8"] Nov 01 23:10:23.418085 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:23.418061 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d73b25-7ffc64f5d5-7wcv8"] Nov 01 23:10:23.748727 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:23.748695 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fee5226-37e2-4c83-9225-449df015af93" path="/var/lib/kubelet/pods/8fee5226-37e2-4c83-9225-449df015af93/volumes" Nov 01 23:10:25.027191 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:25.027157 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:10:29.282074 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.282008 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" containerID="cri-o://2d901a2b05894e46dbfa113a1d03a8c2dd3f2926b2d0bcbcbfbc821682db5071" gracePeriod=45 Nov 01 23:10:29.300945 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.300912 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" containerID="cri-o://56acf032d1ff9cfa5283c04799f6258894976d090fd4e2761aaf844ae04f8c59" gracePeriod=45 Nov 01 23:10:29.456638 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.456566 5011 generic.go:356] "Generic (PLEG): container finished" podID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerID="56acf032d1ff9cfa5283c04799f6258894976d090fd4e2761aaf844ae04f8c59" exitCode=0 Nov 01 23:10:29.457148 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.457119 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" event={"ID":"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0","Type":"ContainerDied","Data":"56acf032d1ff9cfa5283c04799f6258894976d090fd4e2761aaf844ae04f8c59"} Nov 01 23:10:29.486522 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.486477 5011 generic.go:356] "Generic (PLEG): container finished" podID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerID="2d901a2b05894e46dbfa113a1d03a8c2dd3f2926b2d0bcbcbfbc821682db5071" exitCode=0 Nov 01 23:10:29.486694 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.486583 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" event={"ID":"37f3b7d0-5402-4055-82cf-f124eb0a2af6","Type":"ContainerDied","Data":"2d901a2b05894e46dbfa113a1d03a8c2dd3f2926b2d0bcbcbfbc821682db5071"} Nov 01 23:10:29.741737 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.741331 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 23:10:29.753923 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.753893 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 23:10:29.800776 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.800691 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-kube-api-access-2wc2g\" (UniqueName: \"kubernetes.io/projected/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-kube-api-access-2wc2g\") pod \"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0\" (UID: \"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0\") " Nov 01 23:10:29.801494 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.801468 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-home\" (UniqueName: \"kubernetes.io/empty-dir/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-home\") pod \"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0\" (UID: \"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0\") " Nov 01 23:10:29.802096 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.802077 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-tmp-dir\") pod \"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0\" (UID: \"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0\") " Nov 01 23:10:29.802348 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.802329 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-dshm\") pod \"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0\" (UID: \"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0\") " Nov 01 23:10:29.802488 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.802473 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37f3b7d0-5402-4055-82cf-f124eb0a2af6-home\" (UniqueName: \"kubernetes.io/empty-dir/37f3b7d0-5402-4055-82cf-f124eb0a2af6-home\") pod \"37f3b7d0-5402-4055-82cf-f124eb0a2af6\" (UID: \"37f3b7d0-5402-4055-82cf-f124eb0a2af6\") " Nov 01 23:10:29.802626 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.802610 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37f3b7d0-5402-4055-82cf-f124eb0a2af6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/37f3b7d0-5402-4055-82cf-f124eb0a2af6-dshm\") pod \"37f3b7d0-5402-4055-82cf-f124eb0a2af6\" (UID: \"37f3b7d0-5402-4055-82cf-f124eb0a2af6\") " Nov 01 23:10:29.802764 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.802748 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37f3b7d0-5402-4055-82cf-f124eb0a2af6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37f3b7d0-5402-4055-82cf-f124eb0a2af6-model-cache\") pod \"37f3b7d0-5402-4055-82cf-f124eb0a2af6\" (UID: \"37f3b7d0-5402-4055-82cf-f124eb0a2af6\") " Nov 01 23:10:29.802974 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.802956 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37f3b7d0-5402-4055-82cf-f124eb0a2af6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f3b7d0-5402-4055-82cf-f124eb0a2af6-tmp-dir\") pod \"37f3b7d0-5402-4055-82cf-f124eb0a2af6\" (UID: \"37f3b7d0-5402-4055-82cf-f124eb0a2af6\") " Nov 01 23:10:29.803132 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.803117 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-tls-certs\" (UniqueName: \"kubernetes.io/secret/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-tls-certs\") pod \"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0\" (UID: \"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0\") " Nov 01 23:10:29.803533 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.803459 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37f3b7d0-5402-4055-82cf-f124eb0a2af6-kube-api-access-qlgq7\" (UniqueName: \"kubernetes.io/projected/37f3b7d0-5402-4055-82cf-f124eb0a2af6-kube-api-access-qlgq7\") pod \"37f3b7d0-5402-4055-82cf-f124eb0a2af6\" (UID: \"37f3b7d0-5402-4055-82cf-f124eb0a2af6\") " Nov 01 23:10:29.803694 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.803676 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-model-cache\") pod \"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0\" (UID: \"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0\") " Nov 01 23:10:29.803962 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.803943 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37f3b7d0-5402-4055-82cf-f124eb0a2af6-tls-certs\" (UniqueName: \"kubernetes.io/secret/37f3b7d0-5402-4055-82cf-f124eb0a2af6-tls-certs\") pod \"37f3b7d0-5402-4055-82cf-f124eb0a2af6\" (UID: \"37f3b7d0-5402-4055-82cf-f124eb0a2af6\") " Nov 01 23:10:29.806335 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.801892 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-home" pod "7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" (UID: "7a134ecf-28e8-41a3-ac49-e56fa1b93dd0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:29.806335 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.805017 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-kube-api-access-2wc2g" pod "7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" (UID: "7a134ecf-28e8-41a3-ac49-e56fa1b93dd0"). InnerVolumeSpecName "kube-api-access-2wc2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:29.806335 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.806143 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f3b7d0-5402-4055-82cf-f124eb0a2af6-model-cache" pod "37f3b7d0-5402-4055-82cf-f124eb0a2af6" (UID: "37f3b7d0-5402-4055-82cf-f124eb0a2af6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:29.806876 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.806681 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-model-cache" pod "7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" (UID: "7a134ecf-28e8-41a3-ac49-e56fa1b93dd0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:29.807270 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.807014 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f3b7d0-5402-4055-82cf-f124eb0a2af6-tmp-dir" pod "37f3b7d0-5402-4055-82cf-f124eb0a2af6" (UID: "37f3b7d0-5402-4055-82cf-f124eb0a2af6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:29.807602 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.807581 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-tmp-dir" pod "7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" (UID: "7a134ecf-28e8-41a3-ac49-e56fa1b93dd0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:29.807922 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.807903 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f3b7d0-5402-4055-82cf-f124eb0a2af6-home" pod "37f3b7d0-5402-4055-82cf-f124eb0a2af6" (UID: "37f3b7d0-5402-4055-82cf-f124eb0a2af6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:29.813990 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.813959 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-tls-certs" pod "7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" (UID: "7a134ecf-28e8-41a3-ac49-e56fa1b93dd0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:10:29.814134 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.814112 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-dshm" pod "7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" (UID: "7a134ecf-28e8-41a3-ac49-e56fa1b93dd0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:29.837304 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.837266 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37f3b7d0-5402-4055-82cf-f124eb0a2af6-tls-certs" pod "37f3b7d0-5402-4055-82cf-f124eb0a2af6" (UID: "37f3b7d0-5402-4055-82cf-f124eb0a2af6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:10:29.838336 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.838294 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f3b7d0-5402-4055-82cf-f124eb0a2af6-dshm" pod "37f3b7d0-5402-4055-82cf-f124eb0a2af6" (UID: "37f3b7d0-5402-4055-82cf-f124eb0a2af6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:29.841635 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.841582 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f3b7d0-5402-4055-82cf-f124eb0a2af6-kube-api-access-qlgq7" pod "37f3b7d0-5402-4055-82cf-f124eb0a2af6" (UID: "37f3b7d0-5402-4055-82cf-f124eb0a2af6"). InnerVolumeSpecName "kube-api-access-qlgq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:29.907368 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.906885 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-home\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:29.907368 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.906919 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-tmp-dir\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:29.907368 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.906932 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-dshm\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:29.907368 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.906944 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37f3b7d0-5402-4055-82cf-f124eb0a2af6-home\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:29.907368 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.906958 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37f3b7d0-5402-4055-82cf-f124eb0a2af6-dshm\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:29.907874 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.907845 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37f3b7d0-5402-4055-82cf-f124eb0a2af6-model-cache\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:29.907874 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.907877 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f3b7d0-5402-4055-82cf-f124eb0a2af6-tmp-dir\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:29.908083 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.907893 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-tls-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:29.908083 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.907908 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlgq7\" (UniqueName: \"kubernetes.io/projected/37f3b7d0-5402-4055-82cf-f124eb0a2af6-kube-api-access-qlgq7\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:29.908083 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.907922 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-model-cache\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:29.908083 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.907936 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f3b7d0-5402-4055-82cf-f124eb0a2af6-tls-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:29.908083 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:29.907950 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wc2g\" (UniqueName: \"kubernetes.io/projected/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0-kube-api-access-2wc2g\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:30.495188 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:30.495160 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" Nov 01 23:10:30.495678 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:30.495160 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t" event={"ID":"7a134ecf-28e8-41a3-ac49-e56fa1b93dd0","Type":"ContainerDied","Data":"f7e7fd4072e7243a7be4de151a978f773f6c302783a4d67c86bb21dfd8d38c48"} Nov 01 23:10:30.495678 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:30.495296 5011 scope.go:122] "RemoveContainer" containerID="56acf032d1ff9cfa5283c04799f6258894976d090fd4e2761aaf844ae04f8c59" Nov 01 23:10:30.497302 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:30.497280 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" Nov 01 23:10:30.497302 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:30.497292 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn" event={"ID":"37f3b7d0-5402-4055-82cf-f124eb0a2af6","Type":"ContainerDied","Data":"e76c0f1187e465554360d3f9331910c47554d9454589ed4e0d0676c8619701c6"} Nov 01 23:10:30.507690 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:30.507668 5011 scope.go:122] "RemoveContainer" containerID="2d901a2b05894e46dbfa113a1d03a8c2dd3f2926b2d0bcbcbfbc821682db5071" Nov 01 23:10:30.525629 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:30.525597 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn"] Nov 01 23:10:30.533322 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:30.533297 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9ad303/distinct-2-w2-9ad303-kserve-c4f6d4ddb-gt4vn"] Nov 01 23:10:30.546359 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:30.546333 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t"] Nov 01 23:10:30.554796 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:30.554719 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9ad303/simulator-w2-9ad303-kserve-79774d57f9-cs89t"] Nov 01 23:10:31.747694 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:31.747621 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" path="/var/lib/kubelet/pods/37f3b7d0-5402-4055-82cf-f124eb0a2af6/volumes" Nov 01 23:10:31.748077 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:31.748061 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" path="/var/lib/kubelet/pods/7a134ecf-28e8-41a3-ac49-e56fa1b93dd0/volumes" Nov 01 23:10:36.534549 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:36.534516 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq"] Nov 01 23:10:36.535097 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:36.534816 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" podUID="07a622eb-374c-400c-bb23-daccb27e6a3a" containerName="istio-proxy" containerID="cri-o://da6cdb2dea78aa39b5d8e1a10f98d565e646c0e15aa0676e2040731895573163" gracePeriod=30 Nov 01 23:10:37.071080 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:37.071043 5011 patch_prober.go:29] interesting pod/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:10:37.071281 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:37.071114 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" podUID="07a622eb-374c-400c-bb23-daccb27e6a3a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:10:39.778169 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:39.778134 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk"] Nov 01 23:10:39.778689 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:39.778351 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk" podUID="5eef0c69-a918-4275-a819-7aed191fd239" containerName="maas-api" containerID="cri-o://5872f2cf363526cf43730a59bc0ffafa4b89858c0e7dd54bd333288f404973a9" gracePeriod=30 Nov 01 23:10:40.212398 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:40.212370 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk" Nov 01 23:10:40.340412 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:40.340368 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5eef0c69-a918-4275-a819-7aed191fd239-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eef0c69-a918-4275-a819-7aed191fd239-metrics-certs\") pod \"5eef0c69-a918-4275-a819-7aed191fd239\" (UID: \"5eef0c69-a918-4275-a819-7aed191fd239\") " Nov 01 23:10:40.340587 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:40.340437 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5eef0c69-a918-4275-a819-7aed191fd239-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eef0c69-a918-4275-a819-7aed191fd239-maas-api-tls\") pod \"5eef0c69-a918-4275-a819-7aed191fd239\" (UID: \"5eef0c69-a918-4275-a819-7aed191fd239\") " Nov 01 23:10:40.340668 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:40.340636 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eef0c69-a918-4275-a819-7aed191fd239-kube-api-access-k7stt\" (UniqueName: \"kubernetes.io/projected/5eef0c69-a918-4275-a819-7aed191fd239-kube-api-access-k7stt\") pod \"5eef0c69-a918-4275-a819-7aed191fd239\" (UID: \"5eef0c69-a918-4275-a819-7aed191fd239\") " Nov 01 23:10:40.340786 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:40.340755 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eef0c69-a918-4275-a819-7aed191fd239-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eef0c69-a918-4275-a819-7aed191fd239-openshift-service-ca\") pod \"5eef0c69-a918-4275-a819-7aed191fd239\" (UID: \"5eef0c69-a918-4275-a819-7aed191fd239\") " Nov 01 23:10:40.341226 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:40.341201 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eef0c69-a918-4275-a819-7aed191fd239-openshift-service-ca" pod "5eef0c69-a918-4275-a819-7aed191fd239" (UID: "5eef0c69-a918-4275-a819-7aed191fd239"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:40.341549 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:40.341514 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eef0c69-a918-4275-a819-7aed191fd239-openshift-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:40.343159 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:40.343122 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eef0c69-a918-4275-a819-7aed191fd239-metrics-certs" pod "5eef0c69-a918-4275-a819-7aed191fd239" (UID: "5eef0c69-a918-4275-a819-7aed191fd239"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:10:40.349222 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:40.349185 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eef0c69-a918-4275-a819-7aed191fd239-kube-api-access-k7stt" pod "5eef0c69-a918-4275-a819-7aed191fd239" (UID: "5eef0c69-a918-4275-a819-7aed191fd239"). InnerVolumeSpecName "kube-api-access-k7stt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:40.349678 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:40.349624 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eef0c69-a918-4275-a819-7aed191fd239-maas-api-tls" pod "5eef0c69-a918-4275-a819-7aed191fd239" (UID: "5eef0c69-a918-4275-a819-7aed191fd239"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:10:40.443358 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:40.443267 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7stt\" (UniqueName: \"kubernetes.io/projected/5eef0c69-a918-4275-a819-7aed191fd239-kube-api-access-k7stt\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:40.443358 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:40.443295 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eef0c69-a918-4275-a819-7aed191fd239-metrics-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:40.443358 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:40.443305 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eef0c69-a918-4275-a819-7aed191fd239-maas-api-tls\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:40.573950 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:40.573917 5011 generic.go:356] "Generic (PLEG): container finished" podID="5eef0c69-a918-4275-a819-7aed191fd239" containerID="5872f2cf363526cf43730a59bc0ffafa4b89858c0e7dd54bd333288f404973a9" exitCode=0 Nov 01 23:10:40.574125 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:40.573973 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk" Nov 01 23:10:40.574125 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:40.573999 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk" event={"ID":"5eef0c69-a918-4275-a819-7aed191fd239","Type":"ContainerDied","Data":"5872f2cf363526cf43730a59bc0ffafa4b89858c0e7dd54bd333288f404973a9"} Nov 01 23:10:40.574125 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:40.574059 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk" event={"ID":"5eef0c69-a918-4275-a819-7aed191fd239","Type":"ContainerDied","Data":"bda7c41b973389cce8b88cb7fb814ebee737c8041e0fca52cd4413fa9e5c3817"} Nov 01 23:10:40.574125 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:40.574090 5011 scope.go:122] "RemoveContainer" containerID="5872f2cf363526cf43730a59bc0ffafa4b89858c0e7dd54bd333288f404973a9" Nov 01 23:10:40.585817 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:40.585794 5011 scope.go:122] "RemoveContainer" containerID="5872f2cf363526cf43730a59bc0ffafa4b89858c0e7dd54bd333288f404973a9" Nov 01 23:10:40.586176 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:10:40.586154 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5872f2cf363526cf43730a59bc0ffafa4b89858c0e7dd54bd333288f404973a9\": container with ID starting with 5872f2cf363526cf43730a59bc0ffafa4b89858c0e7dd54bd333288f404973a9 not found: ID does not exist" containerID="5872f2cf363526cf43730a59bc0ffafa4b89858c0e7dd54bd333288f404973a9" Nov 01 23:10:40.586231 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:40.586187 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5872f2cf363526cf43730a59bc0ffafa4b89858c0e7dd54bd333288f404973a9"} err="failed to get container status \"5872f2cf363526cf43730a59bc0ffafa4b89858c0e7dd54bd333288f404973a9\": rpc error: code = NotFound desc = could not find container \"5872f2cf363526cf43730a59bc0ffafa4b89858c0e7dd54bd333288f404973a9\": container with ID starting with 5872f2cf363526cf43730a59bc0ffafa4b89858c0e7dd54bd333288f404973a9 not found: ID does not exist" Nov 01 23:10:40.597946 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:40.597916 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk"] Nov 01 23:10:40.602744 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:40.602719 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-9ad303-7d745b4d57-ghjjk"] Nov 01 23:10:41.764131 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:41.764092 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eef0c69-a918-4275-a819-7aed191fd239" path="/var/lib/kubelet/pods/5eef0c69-a918-4275-a819-7aed191fd239/volumes" Nov 01 23:10:41.853439 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:41.853408 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:10:41.967951 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:41.967913 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a622eb-374c-400c-bb23-daccb27e6a3a-istio-token\" (UniqueName: \"kubernetes.io/projected/07a622eb-374c-400c-bb23-daccb27e6a3a-istio-token\") pod \"07a622eb-374c-400c-bb23-daccb27e6a3a\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " Nov 01 23:10:41.968135 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:41.967964 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a622eb-374c-400c-bb23-daccb27e6a3a-kube-api-access-qpbf4\" (UniqueName: \"kubernetes.io/projected/07a622eb-374c-400c-bb23-daccb27e6a3a-kube-api-access-qpbf4\") pod \"07a622eb-374c-400c-bb23-daccb27e6a3a\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " Nov 01 23:10:41.968135 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:41.968021 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-credential-socket\") pod \"07a622eb-374c-400c-bb23-daccb27e6a3a\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " Nov 01 23:10:41.968135 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:41.968099 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-istio-data\") pod \"07a622eb-374c-400c-bb23-daccb27e6a3a\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " Nov 01 23:10:41.968295 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:41.968157 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07a622eb-374c-400c-bb23-daccb27e6a3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a622eb-374c-400c-bb23-daccb27e6a3a-istiod-ca-cert\") pod \"07a622eb-374c-400c-bb23-daccb27e6a3a\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " Nov 01 23:10:41.968295 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:41.968272 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07a622eb-374c-400c-bb23-daccb27e6a3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a622eb-374c-400c-bb23-daccb27e6a3a-istio-podinfo\") pod \"07a622eb-374c-400c-bb23-daccb27e6a3a\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " Nov 01 23:10:41.968417 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:41.968358 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-workload-socket\") pod \"07a622eb-374c-400c-bb23-daccb27e6a3a\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " Nov 01 23:10:41.968472 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:41.968416 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-istio-envoy\") pod \"07a622eb-374c-400c-bb23-daccb27e6a3a\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " Nov 01 23:10:41.968472 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:41.968456 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-workload-certs\") pod \"07a622eb-374c-400c-bb23-daccb27e6a3a\" (UID: \"07a622eb-374c-400c-bb23-daccb27e6a3a\") " Nov 01 23:10:41.968614 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:41.968592 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a622eb-374c-400c-bb23-daccb27e6a3a-istiod-ca-cert" pod "07a622eb-374c-400c-bb23-daccb27e6a3a" (UID: "07a622eb-374c-400c-bb23-daccb27e6a3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:41.968707 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:41.968676 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-workload-socket" pod "07a622eb-374c-400c-bb23-daccb27e6a3a" (UID: "07a622eb-374c-400c-bb23-daccb27e6a3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:41.968898 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:41.968878 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-credential-socket" pod "07a622eb-374c-400c-bb23-daccb27e6a3a" (UID: "07a622eb-374c-400c-bb23-daccb27e6a3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:41.969016 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:41.968929 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-workload-certs" pod "07a622eb-374c-400c-bb23-daccb27e6a3a" (UID: "07a622eb-374c-400c-bb23-daccb27e6a3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:41.969224 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:41.969176 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:41.969224 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:41.969230 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a622eb-374c-400c-bb23-daccb27e6a3a-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:41.969224 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:41.969246 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:41.970480 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:41.969260 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:41.970577 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:41.970553 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-istio-data" pod "07a622eb-374c-400c-bb23-daccb27e6a3a" (UID: "07a622eb-374c-400c-bb23-daccb27e6a3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:41.971387 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:41.971355 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07a622eb-374c-400c-bb23-daccb27e6a3a-istio-podinfo" pod "07a622eb-374c-400c-bb23-daccb27e6a3a" (UID: "07a622eb-374c-400c-bb23-daccb27e6a3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:10:41.972017 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:41.971985 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a622eb-374c-400c-bb23-daccb27e6a3a-istio-token" pod "07a622eb-374c-400c-bb23-daccb27e6a3a" (UID: "07a622eb-374c-400c-bb23-daccb27e6a3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:41.972160 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:41.972139 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-istio-envoy" pod "07a622eb-374c-400c-bb23-daccb27e6a3a" (UID: "07a622eb-374c-400c-bb23-daccb27e6a3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:41.972658 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:41.972631 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a622eb-374c-400c-bb23-daccb27e6a3a-kube-api-access-qpbf4" pod "07a622eb-374c-400c-bb23-daccb27e6a3a" (UID: "07a622eb-374c-400c-bb23-daccb27e6a3a"). InnerVolumeSpecName "kube-api-access-qpbf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:42.074269 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:42.074054 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:42.074269 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:42.074101 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a622eb-374c-400c-bb23-daccb27e6a3a-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:42.074269 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:42.074120 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a622eb-374c-400c-bb23-daccb27e6a3a-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:42.074269 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:42.074140 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a622eb-374c-400c-bb23-daccb27e6a3a-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:42.074269 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:42.074160 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpbf4\" (UniqueName: \"kubernetes.io/projected/07a622eb-374c-400c-bb23-daccb27e6a3a-kube-api-access-qpbf4\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:42.593797 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:42.593763 5011 generic.go:356] "Generic (PLEG): container finished" podID="07a622eb-374c-400c-bb23-daccb27e6a3a" containerID="da6cdb2dea78aa39b5d8e1a10f98d565e646c0e15aa0676e2040731895573163" exitCode=0 Nov 01 23:10:42.594176 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:42.593818 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" Nov 01 23:10:42.594176 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:42.593819 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" event={"ID":"07a622eb-374c-400c-bb23-daccb27e6a3a","Type":"ContainerDied","Data":"da6cdb2dea78aa39b5d8e1a10f98d565e646c0e15aa0676e2040731895573163"} Nov 01 23:10:42.594176 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:42.594001 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq" event={"ID":"07a622eb-374c-400c-bb23-daccb27e6a3a","Type":"ContainerDied","Data":"475604e9a2d21e175ac6702c1e9b75a02f1a0516af7367ad2760ad8df9d768c1"} Nov 01 23:10:42.594176 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:42.594047 5011 scope.go:122] "RemoveContainer" containerID="da6cdb2dea78aa39b5d8e1a10f98d565e646c0e15aa0676e2040731895573163" Nov 01 23:10:42.607302 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:42.607281 5011 scope.go:122] "RemoveContainer" containerID="04f817f300e1124a616af657f8070286bb1ed67c1d73072fd5473dd7c279694b" Nov 01 23:10:42.621050 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:42.621020 5011 scope.go:122] "RemoveContainer" containerID="da6cdb2dea78aa39b5d8e1a10f98d565e646c0e15aa0676e2040731895573163" Nov 01 23:10:42.622268 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:10:42.622223 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da6cdb2dea78aa39b5d8e1a10f98d565e646c0e15aa0676e2040731895573163\": container with ID starting with da6cdb2dea78aa39b5d8e1a10f98d565e646c0e15aa0676e2040731895573163 not found: ID does not exist" containerID="da6cdb2dea78aa39b5d8e1a10f98d565e646c0e15aa0676e2040731895573163" Nov 01 23:10:42.622420 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:42.622269 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da6cdb2dea78aa39b5d8e1a10f98d565e646c0e15aa0676e2040731895573163"} err="failed to get container status \"da6cdb2dea78aa39b5d8e1a10f98d565e646c0e15aa0676e2040731895573163\": rpc error: code = NotFound desc = could not find container \"da6cdb2dea78aa39b5d8e1a10f98d565e646c0e15aa0676e2040731895573163\": container with ID starting with da6cdb2dea78aa39b5d8e1a10f98d565e646c0e15aa0676e2040731895573163 not found: ID does not exist" Nov 01 23:10:42.622420 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:42.622293 5011 scope.go:122] "RemoveContainer" containerID="04f817f300e1124a616af657f8070286bb1ed67c1d73072fd5473dd7c279694b" Nov 01 23:10:42.622765 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:10:42.622721 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04f817f300e1124a616af657f8070286bb1ed67c1d73072fd5473dd7c279694b\": container with ID starting with 04f817f300e1124a616af657f8070286bb1ed67c1d73072fd5473dd7c279694b not found: ID does not exist" containerID="04f817f300e1124a616af657f8070286bb1ed67c1d73072fd5473dd7c279694b" Nov 01 23:10:42.622925 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:42.622763 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04f817f300e1124a616af657f8070286bb1ed67c1d73072fd5473dd7c279694b"} err="failed to get container status \"04f817f300e1124a616af657f8070286bb1ed67c1d73072fd5473dd7c279694b\": rpc error: code = NotFound desc = could not find container \"04f817f300e1124a616af657f8070286bb1ed67c1d73072fd5473dd7c279694b\": container with ID starting with 04f817f300e1124a616af657f8070286bb1ed67c1d73072fd5473dd7c279694b not found: ID does not exist" Nov 01 23:10:42.622925 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:42.622864 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq"] Nov 01 23:10:42.626236 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:42.626211 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-ec0da7-openshift-default-6b7d89bd9-lpzqq"] Nov 01 23:10:43.749239 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:43.749198 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07a622eb-374c-400c-bb23-daccb27e6a3a" path="/var/lib/kubelet/pods/07a622eb-374c-400c-bb23-daccb27e6a3a/volumes" Nov 01 23:10:46.575392 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:46.575354 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z"] Nov 01 23:10:46.576085 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:46.575678 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" podUID="88b7dfae-6aa7-41bf-b479-fb5468f9cc9c" containerName="istio-proxy" containerID="cri-o://ed7c4b5015a66ec471e3da78dc0846c0769e799e8a69c3790f15498bb45c30b0" gracePeriod=30 Nov 01 23:10:46.580869 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:46.580798 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4"] Nov 01 23:10:46.581292 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:46.581062 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" podUID="824b3eb9-831a-4e29-a875-da055383cd7d" containerName="istio-proxy" containerID="cri-o://3740ecf9d40bc15fc5bbc76119758e2e2c85c29612f5265cdbe9bd389e6f108c" gracePeriod=30 Nov 01 23:10:46.973589 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:10:46.973550 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:10:47.201532 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:47.201490 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-c6cec6-5fb65c74cf-gzv7m"] Nov 01 23:10:47.224957 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:47.224866 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c6cec6-5dbc7cdf79-fv8g6"] Nov 01 23:10:47.859935 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:47.859883 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9"] Nov 01 23:10:47.866223 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:47.866187 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" podUID="ed165c04-d90b-45c4-8022-670fb4f2b61c" containerName="istio-proxy" containerID="cri-o://a8b86cb243231103f2239e4ea6e8444455c12504f542687571e3b6a6802b64e4" gracePeriod=30 Nov 01 23:10:51.904721 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:51.904692 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:10:52.048596 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.048544 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-workload-socket" pod "88b7dfae-6aa7-41bf-b479-fb5468f9cc9c" (UID: "88b7dfae-6aa7-41bf-b479-fb5468f9cc9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:52.048791 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.047287 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-workload-socket\") pod \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " Nov 01 23:10:52.048917 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.048856 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-workload-certs\") pod \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " Nov 01 23:10:52.049230 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.049205 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-workload-certs" pod "88b7dfae-6aa7-41bf-b479-fb5468f9cc9c" (UID: "88b7dfae-6aa7-41bf-b479-fb5468f9cc9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:52.050969 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.050941 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istio-token\" (UniqueName: \"kubernetes.io/projected/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istio-token\") pod \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " Nov 01 23:10:52.051094 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.050998 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istiod-ca-cert\") pod \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " Nov 01 23:10:52.051169 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.051122 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-credential-socket\") pod \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " Nov 01 23:10:52.051169 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.051163 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istio-data\") pod \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " Nov 01 23:10:52.051300 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.051200 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istio-podinfo\") pod \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " Nov 01 23:10:52.051300 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.051266 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-kube-api-access-b9d7w\" (UniqueName: \"kubernetes.io/projected/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-kube-api-access-b9d7w\") pod \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " Nov 01 23:10:52.051420 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.051334 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istio-envoy\") pod \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\" (UID: \"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c\") " Nov 01 23:10:52.052202 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.051978 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.052202 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.052000 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.053007 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.052977 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-credential-socket" pod "88b7dfae-6aa7-41bf-b479-fb5468f9cc9c" (UID: "88b7dfae-6aa7-41bf-b479-fb5468f9cc9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:52.053242 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.053217 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istiod-ca-cert" pod "88b7dfae-6aa7-41bf-b479-fb5468f9cc9c" (UID: "88b7dfae-6aa7-41bf-b479-fb5468f9cc9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:52.053341 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.053238 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istio-data" pod "88b7dfae-6aa7-41bf-b479-fb5468f9cc9c" (UID: "88b7dfae-6aa7-41bf-b479-fb5468f9cc9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:52.054865 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.054818 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istio-token" pod "88b7dfae-6aa7-41bf-b479-fb5468f9cc9c" (UID: "88b7dfae-6aa7-41bf-b479-fb5468f9cc9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:52.057255 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.057215 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istio-envoy" pod "88b7dfae-6aa7-41bf-b479-fb5468f9cc9c" (UID: "88b7dfae-6aa7-41bf-b479-fb5468f9cc9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:52.057727 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.057698 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istio-podinfo" pod "88b7dfae-6aa7-41bf-b479-fb5468f9cc9c" (UID: "88b7dfae-6aa7-41bf-b479-fb5468f9cc9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:10:52.058581 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.058556 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:10:52.060565 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.060536 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-kube-api-access-b9d7w" pod "88b7dfae-6aa7-41bf-b479-fb5468f9cc9c" (UID: "88b7dfae-6aa7-41bf-b479-fb5468f9cc9c"). InnerVolumeSpecName "kube-api-access-b9d7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:52.153086 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.152987 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-istio-envoy\") pod \"824b3eb9-831a-4e29-a875-da055383cd7d\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " Nov 01 23:10:52.153086 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.153045 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-workload-certs\") pod \"824b3eb9-831a-4e29-a875-da055383cd7d\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " Nov 01 23:10:52.153359 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.153171 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/824b3eb9-831a-4e29-a875-da055383cd7d-istio-token\" (UniqueName: \"kubernetes.io/projected/824b3eb9-831a-4e29-a875-da055383cd7d-istio-token\") pod \"824b3eb9-831a-4e29-a875-da055383cd7d\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " Nov 01 23:10:52.153359 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.153211 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-workload-socket\") pod \"824b3eb9-831a-4e29-a875-da055383cd7d\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " Nov 01 23:10:52.153359 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.153320 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/824b3eb9-831a-4e29-a875-da055383cd7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/824b3eb9-831a-4e29-a875-da055383cd7d-istio-podinfo\") pod \"824b3eb9-831a-4e29-a875-da055383cd7d\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " Nov 01 23:10:52.153359 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.153355 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/824b3eb9-831a-4e29-a875-da055383cd7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/824b3eb9-831a-4e29-a875-da055383cd7d-istiod-ca-cert\") pod \"824b3eb9-831a-4e29-a875-da055383cd7d\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " Nov 01 23:10:52.153602 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.153392 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/824b3eb9-831a-4e29-a875-da055383cd7d-kube-api-access-n7wdf\" (UniqueName: \"kubernetes.io/projected/824b3eb9-831a-4e29-a875-da055383cd7d-kube-api-access-n7wdf\") pod \"824b3eb9-831a-4e29-a875-da055383cd7d\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " Nov 01 23:10:52.153602 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.153425 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-credential-socket\") pod \"824b3eb9-831a-4e29-a875-da055383cd7d\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " Nov 01 23:10:52.153602 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.153488 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-workload-certs" pod "824b3eb9-831a-4e29-a875-da055383cd7d" (UID: "824b3eb9-831a-4e29-a875-da055383cd7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:52.153602 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.153525 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-istio-data\") pod \"824b3eb9-831a-4e29-a875-da055383cd7d\" (UID: \"824b3eb9-831a-4e29-a875-da055383cd7d\") " Nov 01 23:10:52.154476 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.154243 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.154476 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.154273 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.154476 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.154283 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.154476 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.154292 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9d7w\" (UniqueName: \"kubernetes.io/projected/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-kube-api-access-b9d7w\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.154476 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.154306 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.154476 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.154317 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.154476 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.154324 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.154476 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.154334 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.155076 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.154587 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-istio-data" pod "824b3eb9-831a-4e29-a875-da055383cd7d" (UID: "824b3eb9-831a-4e29-a875-da055383cd7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:52.155452 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.155422 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-credential-socket" pod "824b3eb9-831a-4e29-a875-da055383cd7d" (UID: "824b3eb9-831a-4e29-a875-da055383cd7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:52.155643 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.155488 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-workload-socket" pod "824b3eb9-831a-4e29-a875-da055383cd7d" (UID: "824b3eb9-831a-4e29-a875-da055383cd7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:52.155892 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.155845 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/824b3eb9-831a-4e29-a875-da055383cd7d-istiod-ca-cert" pod "824b3eb9-831a-4e29-a875-da055383cd7d" (UID: "824b3eb9-831a-4e29-a875-da055383cd7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:52.156479 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.156453 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-istio-envoy" pod "824b3eb9-831a-4e29-a875-da055383cd7d" (UID: "824b3eb9-831a-4e29-a875-da055383cd7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:52.157882 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.157853 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/824b3eb9-831a-4e29-a875-da055383cd7d-istio-podinfo" pod "824b3eb9-831a-4e29-a875-da055383cd7d" (UID: "824b3eb9-831a-4e29-a875-da055383cd7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:10:52.158138 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.158113 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824b3eb9-831a-4e29-a875-da055383cd7d-istio-token" pod "824b3eb9-831a-4e29-a875-da055383cd7d" (UID: "824b3eb9-831a-4e29-a875-da055383cd7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:52.158495 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.158475 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824b3eb9-831a-4e29-a875-da055383cd7d-kube-api-access-n7wdf" pod "824b3eb9-831a-4e29-a875-da055383cd7d" (UID: "824b3eb9-831a-4e29-a875-da055383cd7d"). InnerVolumeSpecName "kube-api-access-n7wdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:52.201798 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.201752 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c6cec6-5fb65c74cf-gzv7m" podUID="cc6e9b92-6af6-43b2-b88a-9565806323bc" containerName="payload-processing" containerID="cri-o://5c501b0add172d2f58632d62fe5d1f4d05b807075dea687e75afa49bfc800480" gracePeriod=25 Nov 01 23:10:52.226166 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.226098 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c6cec6-5dbc7cdf79-fv8g6" podUID="39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7" containerName="payload-pre-processing" containerID="cri-o://67b23cc879b660a9eb675764058dac8404c53bd19d65f6f36bbe6f35e62452f2" gracePeriod=25 Nov 01 23:10:52.255545 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.255513 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/824b3eb9-831a-4e29-a875-da055383cd7d-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.255545 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.255541 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/824b3eb9-831a-4e29-a875-da055383cd7d-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.255730 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.255552 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7wdf\" (UniqueName: \"kubernetes.io/projected/824b3eb9-831a-4e29-a875-da055383cd7d-kube-api-access-n7wdf\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.255730 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.255563 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.255730 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.255571 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.255730 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.255580 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.255730 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.255588 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/824b3eb9-831a-4e29-a875-da055383cd7d-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.255730 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.255596 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/824b3eb9-831a-4e29-a875-da055383cd7d-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.670971 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.670934 5011 generic.go:356] "Generic (PLEG): container finished" podID="88b7dfae-6aa7-41bf-b479-fb5468f9cc9c" containerID="ed7c4b5015a66ec471e3da78dc0846c0769e799e8a69c3790f15498bb45c30b0" exitCode=0 Nov 01 23:10:52.671129 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.670984 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" Nov 01 23:10:52.671129 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.671013 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" event={"ID":"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c","Type":"ContainerDied","Data":"ed7c4b5015a66ec471e3da78dc0846c0769e799e8a69c3790f15498bb45c30b0"} Nov 01 23:10:52.671129 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.671070 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z" event={"ID":"88b7dfae-6aa7-41bf-b479-fb5468f9cc9c","Type":"ContainerDied","Data":"d90cfaf7994fcb081e1c8830ea20fc130aaf06c2e04e55dcfcfe8cdc847aad4d"} Nov 01 23:10:52.671129 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.671099 5011 scope.go:122] "RemoveContainer" containerID="ed7c4b5015a66ec471e3da78dc0846c0769e799e8a69c3790f15498bb45c30b0" Nov 01 23:10:52.672706 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.672688 5011 generic.go:356] "Generic (PLEG): container finished" podID="824b3eb9-831a-4e29-a875-da055383cd7d" containerID="3740ecf9d40bc15fc5bbc76119758e2e2c85c29612f5265cdbe9bd389e6f108c" exitCode=0 Nov 01 23:10:52.672811 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.672771 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" event={"ID":"824b3eb9-831a-4e29-a875-da055383cd7d","Type":"ContainerDied","Data":"3740ecf9d40bc15fc5bbc76119758e2e2c85c29612f5265cdbe9bd389e6f108c"} Nov 01 23:10:52.672811 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.672808 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" event={"ID":"824b3eb9-831a-4e29-a875-da055383cd7d","Type":"ContainerDied","Data":"5047c0f0ec649a619ef3f53974ef4b82b88f62501afcf7507314a54eeaeb5f2f"} Nov 01 23:10:52.672940 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.672849 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4" Nov 01 23:10:52.684185 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.684168 5011 scope.go:122] "RemoveContainer" containerID="ed7c4b5015a66ec471e3da78dc0846c0769e799e8a69c3790f15498bb45c30b0" Nov 01 23:10:52.684498 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:10:52.684460 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed7c4b5015a66ec471e3da78dc0846c0769e799e8a69c3790f15498bb45c30b0\": container with ID starting with ed7c4b5015a66ec471e3da78dc0846c0769e799e8a69c3790f15498bb45c30b0 not found: ID does not exist" containerID="ed7c4b5015a66ec471e3da78dc0846c0769e799e8a69c3790f15498bb45c30b0" Nov 01 23:10:52.684594 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.684491 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed7c4b5015a66ec471e3da78dc0846c0769e799e8a69c3790f15498bb45c30b0"} err="failed to get container status \"ed7c4b5015a66ec471e3da78dc0846c0769e799e8a69c3790f15498bb45c30b0\": rpc error: code = NotFound desc = could not find container \"ed7c4b5015a66ec471e3da78dc0846c0769e799e8a69c3790f15498bb45c30b0\": container with ID starting with ed7c4b5015a66ec471e3da78dc0846c0769e799e8a69c3790f15498bb45c30b0 not found: ID does not exist" Nov 01 23:10:52.684594 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.684515 5011 scope.go:122] "RemoveContainer" containerID="3740ecf9d40bc15fc5bbc76119758e2e2c85c29612f5265cdbe9bd389e6f108c" Nov 01 23:10:52.694522 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.694347 5011 scope.go:122] "RemoveContainer" containerID="3740ecf9d40bc15fc5bbc76119758e2e2c85c29612f5265cdbe9bd389e6f108c" Nov 01 23:10:52.694698 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:10:52.694679 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3740ecf9d40bc15fc5bbc76119758e2e2c85c29612f5265cdbe9bd389e6f108c\": container with ID starting with 3740ecf9d40bc15fc5bbc76119758e2e2c85c29612f5265cdbe9bd389e6f108c not found: ID does not exist" containerID="3740ecf9d40bc15fc5bbc76119758e2e2c85c29612f5265cdbe9bd389e6f108c" Nov 01 23:10:52.694740 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.694707 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3740ecf9d40bc15fc5bbc76119758e2e2c85c29612f5265cdbe9bd389e6f108c"} err="failed to get container status \"3740ecf9d40bc15fc5bbc76119758e2e2c85c29612f5265cdbe9bd389e6f108c\": rpc error: code = NotFound desc = could not find container \"3740ecf9d40bc15fc5bbc76119758e2e2c85c29612f5265cdbe9bd389e6f108c\": container with ID starting with 3740ecf9d40bc15fc5bbc76119758e2e2c85c29612f5265cdbe9bd389e6f108c not found: ID does not exist" Nov 01 23:10:52.696295 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.696275 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z"] Nov 01 23:10:52.700938 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.700916 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-j9m6z"] Nov 01 23:10:52.711770 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.711750 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4"] Nov 01 23:10:52.716344 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.716321 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-nwkm4"] Nov 01 23:10:52.764346 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.764314 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg"] Nov 01 23:10:52.764920 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.764898 5011 cpu_manager.go:414] "containerMap: removing container" podUID="d7b301bf-5694-40b2-8789-07555de1ed0f" containerName="istio-proxy" Nov 01 23:10:52.764920 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.764915 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7b301bf-5694-40b2-8789-07555de1ed0f" containerName="istio-proxy" Nov 01 23:10:52.764920 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.764924 5011 cpu_manager.go:414] "containerMap: removing container" podUID="07a622eb-374c-400c-bb23-daccb27e6a3a" containerName="istio-proxy" Nov 01 23:10:52.765101 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.764928 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a622eb-374c-400c-bb23-daccb27e6a3a" containerName="istio-proxy" Nov 01 23:10:52.765101 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.764942 5011 cpu_manager.go:414] "containerMap: removing container" podUID="824b3eb9-831a-4e29-a875-da055383cd7d" containerName="istio-proxy" Nov 01 23:10:52.765101 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.764947 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="824b3eb9-831a-4e29-a875-da055383cd7d" containerName="istio-proxy" Nov 01 23:10:52.765101 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.764959 5011 cpu_manager.go:414] "containerMap: removing container" podUID="be85990a-36c7-4d36-8867-28b4c83867d8" containerName="istio-proxy" Nov 01 23:10:52.765101 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.764964 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be85990a-36c7-4d36-8867-28b4c83867d8" containerName="istio-proxy" Nov 01 23:10:52.765101 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.764972 5011 cpu_manager.go:414] "containerMap: removing container" podUID="88b7dfae-6aa7-41bf-b479-fb5468f9cc9c" containerName="istio-proxy" Nov 01 23:10:52.765101 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.764977 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88b7dfae-6aa7-41bf-b479-fb5468f9cc9c" containerName="istio-proxy" Nov 01 23:10:52.765101 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.764986 5011 cpu_manager.go:414] "containerMap: removing container" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" Nov 01 23:10:52.765101 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.764991 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" Nov 01 23:10:52.765101 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.765000 5011 cpu_manager.go:414] "containerMap: removing container" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" Nov 01 23:10:52.765101 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.765004 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" Nov 01 23:10:52.765101 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.765011 5011 cpu_manager.go:414] "containerMap: removing container" podUID="8fee5226-37e2-4c83-9225-449df015af93" containerName="payload-processing" Nov 01 23:10:52.765101 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.765015 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fee5226-37e2-4c83-9225-449df015af93" containerName="payload-processing" Nov 01 23:10:52.765101 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.765026 5011 cpu_manager.go:414] "containerMap: removing container" podUID="d7b301bf-5694-40b2-8789-07555de1ed0f" containerName="istio-proxy" Nov 01 23:10:52.765101 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.765030 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7b301bf-5694-40b2-8789-07555de1ed0f" containerName="istio-proxy" Nov 01 23:10:52.765101 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.765038 5011 cpu_manager.go:414] "containerMap: removing container" podUID="bc7cb565-6bf1-400f-9a3d-f062af8270bd" containerName="istio-proxy" Nov 01 23:10:52.765101 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.765042 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc7cb565-6bf1-400f-9a3d-f062af8270bd" containerName="istio-proxy" Nov 01 23:10:52.765101 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.765052 5011 cpu_manager.go:414] "containerMap: removing container" podUID="a03efd24-da7e-407d-896d-d22e46206274" containerName="istio-proxy" Nov 01 23:10:52.765101 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.765056 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a03efd24-da7e-407d-896d-d22e46206274" containerName="istio-proxy" Nov 01 23:10:52.765101 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.765068 5011 cpu_manager.go:414] "containerMap: removing container" podUID="0ee11042-8133-4e65-950e-224a5b4ca910" containerName="istio-proxy" Nov 01 23:10:52.765101 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.765073 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ee11042-8133-4e65-950e-224a5b4ca910" containerName="istio-proxy" Nov 01 23:10:52.765101 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.765082 5011 cpu_manager.go:414] "containerMap: removing container" podUID="5eef0c69-a918-4275-a819-7aed191fd239" containerName="maas-api" Nov 01 23:10:52.765101 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.765087 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eef0c69-a918-4275-a819-7aed191fd239" containerName="maas-api" Nov 01 23:10:52.765962 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.765164 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="be85990a-36c7-4d36-8867-28b4c83867d8" containerName="istio-proxy" Nov 01 23:10:52.765962 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.765173 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a622eb-374c-400c-bb23-daccb27e6a3a" containerName="istio-proxy" Nov 01 23:10:52.765962 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.765178 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ee11042-8133-4e65-950e-224a5b4ca910" containerName="istio-proxy" Nov 01 23:10:52.765962 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.765184 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eef0c69-a918-4275-a819-7aed191fd239" containerName="maas-api" Nov 01 23:10:52.765962 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.765191 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7b301bf-5694-40b2-8789-07555de1ed0f" containerName="istio-proxy" Nov 01 23:10:52.765962 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.765196 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a622eb-374c-400c-bb23-daccb27e6a3a" containerName="istio-proxy" Nov 01 23:10:52.765962 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.765202 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="824b3eb9-831a-4e29-a875-da055383cd7d" containerName="istio-proxy" Nov 01 23:10:52.765962 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.765209 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="be85990a-36c7-4d36-8867-28b4c83867d8" containerName="istio-proxy" Nov 01 23:10:52.765962 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.765216 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="88b7dfae-6aa7-41bf-b479-fb5468f9cc9c" containerName="istio-proxy" Nov 01 23:10:52.765962 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.765222 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="a03efd24-da7e-407d-896d-d22e46206274" containerName="istio-proxy" Nov 01 23:10:52.765962 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.765228 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a134ecf-28e8-41a3-ac49-e56fa1b93dd0" containerName="main" Nov 01 23:10:52.765962 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.765234 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="37f3b7d0-5402-4055-82cf-f124eb0a2af6" containerName="main" Nov 01 23:10:52.765962 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.765239 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fee5226-37e2-4c83-9225-449df015af93" containerName="payload-processing" Nov 01 23:10:52.770237 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.770221 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:10:52.782089 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.782060 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg"] Nov 01 23:10:52.966329 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.966283 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-credential-socket\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:10:52.966806 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.966407 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-workload-certs\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:10:52.966806 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.966488 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-workload-socket\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:10:52.966806 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.966590 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6h5b\" (UniqueName: \"kubernetes.io/projected/80ac6525-b39e-4774-b781-0b07dc22f7ab-kube-api-access-z6h5b\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:10:52.966806 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.966742 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80ac6525-b39e-4774-b781-0b07dc22f7ab-istiod-ca-cert\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:10:52.966806 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.966797 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80ac6525-b39e-4774-b781-0b07dc22f7ab-istio-token\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:10:52.967053 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.966932 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-istio-data\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:10:52.967053 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.966974 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80ac6525-b39e-4774-b781-0b07dc22f7ab-istio-podinfo\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:10:52.967127 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:52.967071 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-istio-envoy\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:10:53.068910 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.068871 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80ac6525-b39e-4774-b781-0b07dc22f7ab-istio-token\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:10:53.069094 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.068957 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-istio-data\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:10:53.069094 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.068993 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80ac6525-b39e-4774-b781-0b07dc22f7ab-istio-podinfo\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:10:53.069094 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.069045 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-istio-envoy\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:10:53.069094 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.069074 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-credential-socket\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:10:53.069403 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.069128 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-workload-certs\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:10:53.069403 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.069185 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-workload-socket\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:10:53.069403 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.069228 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6h5b\" (UniqueName: \"kubernetes.io/projected/80ac6525-b39e-4774-b781-0b07dc22f7ab-kube-api-access-z6h5b\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:10:53.070322 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.070292 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80ac6525-b39e-4774-b781-0b07dc22f7ab-istiod-ca-cert\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:10:53.071217 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.071190 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80ac6525-b39e-4774-b781-0b07dc22f7ab-istiod-ca-cert\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:10:53.071796 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.071772 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-istio-data\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:10:53.072752 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.072511 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-credential-socket\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:10:53.072752 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.072606 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-workload-socket\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:10:53.072752 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.072721 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-workload-certs\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:10:53.074948 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.074903 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80ac6525-b39e-4774-b781-0b07dc22f7ab-istio-podinfo\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:10:53.076268 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.076245 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-istio-envoy\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:10:53.080034 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.080012 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80ac6525-b39e-4774-b781-0b07dc22f7ab-istio-token\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:10:53.083874 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.080711 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6h5b\" (UniqueName: \"kubernetes.io/projected/80ac6525-b39e-4774-b781-0b07dc22f7ab-kube-api-access-z6h5b\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:10:53.086618 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.086379 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:10:53.163151 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.163125 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 23:10:53.273324 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.273287 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-workload-socket\") pod \"ed165c04-d90b-45c4-8022-670fb4f2b61c\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " Nov 01 23:10:53.273493 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.273460 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-istio-data\") pod \"ed165c04-d90b-45c4-8022-670fb4f2b61c\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " Nov 01 23:10:53.273541 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.273501 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-workload-certs\") pod \"ed165c04-d90b-45c4-8022-670fb4f2b61c\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " Nov 01 23:10:53.273541 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.273518 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-istio-envoy\") pod \"ed165c04-d90b-45c4-8022-670fb4f2b61c\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " Nov 01 23:10:53.273615 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.273544 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-workload-socket" pod "ed165c04-d90b-45c4-8022-670fb4f2b61c" (UID: "ed165c04-d90b-45c4-8022-670fb4f2b61c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:53.273615 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.273563 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed165c04-d90b-45c4-8022-670fb4f2b61c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed165c04-d90b-45c4-8022-670fb4f2b61c-istiod-ca-cert\") pod \"ed165c04-d90b-45c4-8022-670fb4f2b61c\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " Nov 01 23:10:53.273615 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.273611 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed165c04-d90b-45c4-8022-670fb4f2b61c-istio-token\" (UniqueName: \"kubernetes.io/projected/ed165c04-d90b-45c4-8022-670fb4f2b61c-istio-token\") pod \"ed165c04-d90b-45c4-8022-670fb4f2b61c\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " Nov 01 23:10:53.273723 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.273690 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed165c04-d90b-45c4-8022-670fb4f2b61c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed165c04-d90b-45c4-8022-670fb4f2b61c-istio-podinfo\") pod \"ed165c04-d90b-45c4-8022-670fb4f2b61c\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " Nov 01 23:10:53.273767 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.273721 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed165c04-d90b-45c4-8022-670fb4f2b61c-kube-api-access-w7gkv\" (UniqueName: \"kubernetes.io/projected/ed165c04-d90b-45c4-8022-670fb4f2b61c-kube-api-access-w7gkv\") pod \"ed165c04-d90b-45c4-8022-670fb4f2b61c\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " Nov 01 23:10:53.273767 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.273755 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-credential-socket\") pod \"ed165c04-d90b-45c4-8022-670fb4f2b61c\" (UID: \"ed165c04-d90b-45c4-8022-670fb4f2b61c\") " Nov 01 23:10:53.273890 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.273849 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-istio-data" pod "ed165c04-d90b-45c4-8022-670fb4f2b61c" (UID: "ed165c04-d90b-45c4-8022-670fb4f2b61c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:53.273890 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.273868 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-workload-certs" pod "ed165c04-d90b-45c4-8022-670fb4f2b61c" (UID: "ed165c04-d90b-45c4-8022-670fb4f2b61c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:53.274165 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.274139 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-credential-socket" pod "ed165c04-d90b-45c4-8022-670fb4f2b61c" (UID: "ed165c04-d90b-45c4-8022-670fb4f2b61c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:53.274266 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.274215 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed165c04-d90b-45c4-8022-670fb4f2b61c-istiod-ca-cert" pod "ed165c04-d90b-45c4-8022-670fb4f2b61c" (UID: "ed165c04-d90b-45c4-8022-670fb4f2b61c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:53.274549 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.274489 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:53.274549 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.274520 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:53.274549 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.274535 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:53.274549 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.274550 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:53.274549 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.274563 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed165c04-d90b-45c4-8022-670fb4f2b61c-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:53.276081 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.276059 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-istio-envoy" pod "ed165c04-d90b-45c4-8022-670fb4f2b61c" (UID: "ed165c04-d90b-45c4-8022-670fb4f2b61c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:53.276189 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.276106 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed165c04-d90b-45c4-8022-670fb4f2b61c-istio-podinfo" pod "ed165c04-d90b-45c4-8022-670fb4f2b61c" (UID: "ed165c04-d90b-45c4-8022-670fb4f2b61c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:10:53.276189 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.276139 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed165c04-d90b-45c4-8022-670fb4f2b61c-istio-token" pod "ed165c04-d90b-45c4-8022-670fb4f2b61c" (UID: "ed165c04-d90b-45c4-8022-670fb4f2b61c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:53.276443 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.276428 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed165c04-d90b-45c4-8022-670fb4f2b61c-kube-api-access-w7gkv" pod "ed165c04-d90b-45c4-8022-670fb4f2b61c" (UID: "ed165c04-d90b-45c4-8022-670fb4f2b61c"). InnerVolumeSpecName "kube-api-access-w7gkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:53.375876 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.375811 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed165c04-d90b-45c4-8022-670fb4f2b61c-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:53.375876 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.375875 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed165c04-d90b-45c4-8022-670fb4f2b61c-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:53.376095 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.375888 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed165c04-d90b-45c4-8022-670fb4f2b61c-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:53.376095 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.375897 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7gkv\" (UniqueName: \"kubernetes.io/projected/ed165c04-d90b-45c4-8022-670fb4f2b61c-kube-api-access-w7gkv\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:53.406242 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.406215 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg"] Nov 01 23:10:53.407872 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:10:53.407811 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ac6525_b39e_4774_b781_0b07dc22f7ab.slice/crio-1a879747390a79bdcde8579095c6a3369a3fecbc293e935c8413e19d58002896 WatchSource:0}: Error finding container 1a879747390a79bdcde8579095c6a3369a3fecbc293e935c8413e19d58002896: Status 404 returned error can't find the container with id 1a879747390a79bdcde8579095c6a3369a3fecbc293e935c8413e19d58002896 Nov 01 23:10:53.410437 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.410398 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:10:53.410537 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.410486 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:10:53.410537 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.410525 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:10:53.682710 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.682673 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" event={"ID":"80ac6525-b39e-4774-b781-0b07dc22f7ab","Type":"ContainerStarted","Data":"a7750b1025bec3dbe324e2a07edadd027607b8b9d898af3d17d085f88554cf25"} Nov 01 23:10:53.682969 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.682743 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" event={"ID":"80ac6525-b39e-4774-b781-0b07dc22f7ab","Type":"ContainerStarted","Data":"1a879747390a79bdcde8579095c6a3369a3fecbc293e935c8413e19d58002896"} Nov 01 23:10:53.686735 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.686706 5011 generic.go:356] "Generic (PLEG): container finished" podID="ed165c04-d90b-45c4-8022-670fb4f2b61c" containerID="a8b86cb243231103f2239e4ea6e8444455c12504f542687571e3b6a6802b64e4" exitCode=0 Nov 01 23:10:53.686881 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.686753 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" Nov 01 23:10:53.686881 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.686755 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" event={"ID":"ed165c04-d90b-45c4-8022-670fb4f2b61c","Type":"ContainerDied","Data":"a8b86cb243231103f2239e4ea6e8444455c12504f542687571e3b6a6802b64e4"} Nov 01 23:10:53.686881 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.686849 5011 scope.go:122] "RemoveContainer" containerID="a8b86cb243231103f2239e4ea6e8444455c12504f542687571e3b6a6802b64e4" Nov 01 23:10:53.686881 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.686824 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9" event={"ID":"ed165c04-d90b-45c4-8022-670fb4f2b61c","Type":"ContainerDied","Data":"616c565465bded9588129389444cf753bcbf1ad18d5a76dad93b13221bc99c63"} Nov 01 23:10:53.700015 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.699739 5011 scope.go:122] "RemoveContainer" containerID="a8b86cb243231103f2239e4ea6e8444455c12504f542687571e3b6a6802b64e4" Nov 01 23:10:53.700104 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:10:53.700068 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8b86cb243231103f2239e4ea6e8444455c12504f542687571e3b6a6802b64e4\": container with ID starting with a8b86cb243231103f2239e4ea6e8444455c12504f542687571e3b6a6802b64e4 not found: ID does not exist" containerID="a8b86cb243231103f2239e4ea6e8444455c12504f542687571e3b6a6802b64e4" Nov 01 23:10:53.700104 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.700093 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8b86cb243231103f2239e4ea6e8444455c12504f542687571e3b6a6802b64e4"} err="failed to get container status \"a8b86cb243231103f2239e4ea6e8444455c12504f542687571e3b6a6802b64e4\": rpc error: code = NotFound desc = could not find container \"a8b86cb243231103f2239e4ea6e8444455c12504f542687571e3b6a6802b64e4\": container with ID starting with a8b86cb243231103f2239e4ea6e8444455c12504f542687571e3b6a6802b64e4 not found: ID does not exist" Nov 01 23:10:53.722328 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.722276 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" podStartSLOduration=1.722261518 podStartE2EDuration="1.722261518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:10:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:10:53.719321084 +0000 UTC m=+2822.123911715" watchObservedRunningTime="2026-11-01 23:10:53.722261518 +0000 UTC m=+2822.126852149" Nov 01 23:10:53.749308 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.749267 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="824b3eb9-831a-4e29-a875-da055383cd7d" path="/var/lib/kubelet/pods/824b3eb9-831a-4e29-a875-da055383cd7d/volumes" Nov 01 23:10:53.750099 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.750074 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88b7dfae-6aa7-41bf-b479-fb5468f9cc9c" path="/var/lib/kubelet/pods/88b7dfae-6aa7-41bf-b479-fb5468f9cc9c/volumes" Nov 01 23:10:53.750531 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.750515 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9"] Nov 01 23:10:53.754776 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:53.754751 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-j28j9"] Nov 01 23:10:54.087699 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:54.087607 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:10:54.089504 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:54.089476 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Nov 01 23:10:54.089658 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:54.089533 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" podUID="80ac6525-b39e-4774-b781-0b07dc22f7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Nov 01 23:10:55.088131 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:55.088102 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Nov 01 23:10:55.088496 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:55.088155 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" podUID="80ac6525-b39e-4774-b781-0b07dc22f7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Nov 01 23:10:55.326433 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:55.326401 5011 scope.go:122] "RemoveContainer" containerID="0f721e68026731f516e1add2e3077d144ae4543a65ba259e342644a769a4ffd9" Nov 01 23:10:55.771262 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:55.771230 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed165c04-d90b-45c4-8022-670fb4f2b61c" path="/var/lib/kubelet/pods/ed165c04-d90b-45c4-8022-670fb4f2b61c/volumes" Nov 01 23:10:56.088097 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:56.088017 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Nov 01 23:10:56.088097 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:56.088086 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" podUID="80ac6525-b39e-4774-b781-0b07dc22f7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Nov 01 23:10:57.087274 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:57.087235 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Nov 01 23:10:57.087475 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:57.087294 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" podUID="80ac6525-b39e-4774-b781-0b07dc22f7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Nov 01 23:10:58.087669 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:58.087631 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Nov 01 23:10:58.088098 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:58.087692 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" podUID="80ac6525-b39e-4774-b781-0b07dc22f7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Nov 01 23:10:59.087243 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:59.087210 5011 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Nov 01 23:10:59.087419 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:10:59.087265 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" podUID="80ac6525-b39e-4774-b781-0b07dc22f7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Nov 01 23:11:00.170211 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:00.170174 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:11:00.170775 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:00.170754 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:11:00.171958 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:00.171934 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:11:01.218476 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:01.218423 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c6cec6-5dbc7cdf79-fv8g6" podUID="39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:11:01.218932 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:01.218436 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c6cec6-5fb65c74cf-gzv7m" podUID="cc6e9b92-6af6-43b2-b88a-9565806323bc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:11:02.742854 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:02.741323 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-46ce02-6c8b5f89b8-wr5pn"] Nov 01 23:11:02.742854 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:02.742030 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-6c8b5f89b8-wr5pn" podUID="e3a83ece-8434-49fc-a01f-2f926936384d" containerName="payload-processing" containerID="cri-o://3daecde9e54598133e5820c8f7f4b431f8d973668e70a539cbfd8c39072bb218" gracePeriod=30 Nov 01 23:11:03.220853 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:03.206056 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-6c8b5f89b8-wr5pn" Nov 01 23:11:03.370043 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:03.370000 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3a83ece-8434-49fc-a01f-2f926936384d-kube-api-access-ptjbx\" (UniqueName: \"kubernetes.io/projected/e3a83ece-8434-49fc-a01f-2f926936384d-kube-api-access-ptjbx\") pod \"e3a83ece-8434-49fc-a01f-2f926936384d\" (UID: \"e3a83ece-8434-49fc-a01f-2f926936384d\") " Nov 01 23:11:03.370226 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:03.370073 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3a83ece-8434-49fc-a01f-2f926936384d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3a83ece-8434-49fc-a01f-2f926936384d-plugins-config-volume\") pod \"e3a83ece-8434-49fc-a01f-2f926936384d\" (UID: \"e3a83ece-8434-49fc-a01f-2f926936384d\") " Nov 01 23:11:03.370458 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:03.370432 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3a83ece-8434-49fc-a01f-2f926936384d-plugins-config-volume" pod "e3a83ece-8434-49fc-a01f-2f926936384d" (UID: "e3a83ece-8434-49fc-a01f-2f926936384d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:03.370793 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:03.370766 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3a83ece-8434-49fc-a01f-2f926936384d-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:03.372543 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:03.372514 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3a83ece-8434-49fc-a01f-2f926936384d-kube-api-access-ptjbx" pod "e3a83ece-8434-49fc-a01f-2f926936384d" (UID: "e3a83ece-8434-49fc-a01f-2f926936384d"). InnerVolumeSpecName "kube-api-access-ptjbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:03.472115 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:03.472069 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptjbx\" (UniqueName: \"kubernetes.io/projected/e3a83ece-8434-49fc-a01f-2f926936384d-kube-api-access-ptjbx\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:03.779307 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:03.779260 5011 generic.go:356] "Generic (PLEG): container finished" podID="e3a83ece-8434-49fc-a01f-2f926936384d" containerID="3daecde9e54598133e5820c8f7f4b431f8d973668e70a539cbfd8c39072bb218" exitCode=0 Nov 01 23:11:03.779767 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:03.779318 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-6c8b5f89b8-wr5pn" Nov 01 23:11:03.779767 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:03.779440 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-6c8b5f89b8-wr5pn" event={"ID":"e3a83ece-8434-49fc-a01f-2f926936384d","Type":"ContainerDied","Data":"3daecde9e54598133e5820c8f7f4b431f8d973668e70a539cbfd8c39072bb218"} Nov 01 23:11:03.779767 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:03.779478 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-46ce02-6c8b5f89b8-wr5pn" event={"ID":"e3a83ece-8434-49fc-a01f-2f926936384d","Type":"ContainerDied","Data":"e6148c18e891a8d84444cafcde0c6b5ea69a2554195b184555b5d248ca4e460d"} Nov 01 23:11:03.779767 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:03.779509 5011 scope.go:122] "RemoveContainer" containerID="3daecde9e54598133e5820c8f7f4b431f8d973668e70a539cbfd8c39072bb218" Nov 01 23:11:03.796218 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:03.796142 5011 scope.go:122] "RemoveContainer" containerID="3daecde9e54598133e5820c8f7f4b431f8d973668e70a539cbfd8c39072bb218" Nov 01 23:11:03.796587 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:11:03.796556 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3daecde9e54598133e5820c8f7f4b431f8d973668e70a539cbfd8c39072bb218\": container with ID starting with 3daecde9e54598133e5820c8f7f4b431f8d973668e70a539cbfd8c39072bb218 not found: ID does not exist" containerID="3daecde9e54598133e5820c8f7f4b431f8d973668e70a539cbfd8c39072bb218" Nov 01 23:11:03.796679 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:03.796596 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3daecde9e54598133e5820c8f7f4b431f8d973668e70a539cbfd8c39072bb218"} err="failed to get container status \"3daecde9e54598133e5820c8f7f4b431f8d973668e70a539cbfd8c39072bb218\": rpc error: code = NotFound desc = could not find container \"3daecde9e54598133e5820c8f7f4b431f8d973668e70a539cbfd8c39072bb218\": container with ID starting with 3daecde9e54598133e5820c8f7f4b431f8d973668e70a539cbfd8c39072bb218 not found: ID does not exist" Nov 01 23:11:03.811918 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:03.811887 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-46ce02-6c8b5f89b8-wr5pn"] Nov 01 23:11:03.820134 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:03.819798 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-46ce02-6c8b5f89b8-wr5pn"] Nov 01 23:11:05.784353 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:05.784296 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3a83ece-8434-49fc-a01f-2f926936384d" path="/var/lib/kubelet/pods/e3a83ece-8434-49fc-a01f-2f926936384d/volumes" Nov 01 23:11:10.037565 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:10.037528 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245"] Nov 01 23:11:10.038103 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:10.037754 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" podUID="d295266d-5d50-4aae-a0ee-d86513a36f34" containerName="istio-proxy" containerID="cri-o://f545ecb3a6ae9c637aa4de7689c261e1f5b9833cd6d45d0a6ebf21c12ad15b8d" gracePeriod=30 Nov 01 23:11:11.218102 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:11.218062 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c6cec6-5fb65c74cf-gzv7m" podUID="cc6e9b92-6af6-43b2-b88a-9565806323bc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:11:11.218533 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:11.218102 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c6cec6-5dbc7cdf79-fv8g6" podUID="39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:11:11.387429 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:11.387385 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:11:13.008610 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:13.007356 5011 generic.go:356] "Generic (PLEG): container finished" podID="cc6e9b92-6af6-43b2-b88a-9565806323bc" containerID="5c501b0add172d2f58632d62fe5d1f4d05b807075dea687e75afa49bfc800480" exitCode=0 Nov 01 23:11:13.008610 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:13.007507 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c6cec6-5fb65c74cf-gzv7m" event={"ID":"cc6e9b92-6af6-43b2-b88a-9565806323bc","Type":"ContainerDied","Data":"5c501b0add172d2f58632d62fe5d1f4d05b807075dea687e75afa49bfc800480"} Nov 01 23:11:13.011772 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:13.011547 5011 generic.go:356] "Generic (PLEG): container finished" podID="39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7" containerID="67b23cc879b660a9eb675764058dac8404c53bd19d65f6f36bbe6f35e62452f2" exitCode=0 Nov 01 23:11:13.011772 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:13.011672 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c6cec6-5dbc7cdf79-fv8g6" event={"ID":"39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7","Type":"ContainerDied","Data":"67b23cc879b660a9eb675764058dac8404c53bd19d65f6f36bbe6f35e62452f2"} Nov 01 23:11:13.072729 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:13.072657 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c6cec6-5dbc7cdf79-fv8g6" Nov 01 23:11:13.155984 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:13.155938 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7-plugins-config-volume\") pod \"39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7\" (UID: \"39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7\") " Nov 01 23:11:13.156172 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:13.156017 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7-kube-api-access-64lvs\" (UniqueName: \"kubernetes.io/projected/39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7-kube-api-access-64lvs\") pod \"39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7\" (UID: \"39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7\") " Nov 01 23:11:13.160910 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:13.160825 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7-plugins-config-volume" pod "39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7" (UID: "39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:13.161560 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:13.161531 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7-kube-api-access-64lvs" pod "39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7" (UID: "39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7"). InnerVolumeSpecName "kube-api-access-64lvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:13.190933 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:13.190899 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c6cec6-5fb65c74cf-gzv7m" Nov 01 23:11:13.257560 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:13.257524 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:13.257560 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:13.257559 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64lvs\" (UniqueName: \"kubernetes.io/projected/39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7-kube-api-access-64lvs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:13.359035 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:13.358942 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc6e9b92-6af6-43b2-b88a-9565806323bc-kube-api-access-bbpt4\" (UniqueName: \"kubernetes.io/projected/cc6e9b92-6af6-43b2-b88a-9565806323bc-kube-api-access-bbpt4\") pod \"cc6e9b92-6af6-43b2-b88a-9565806323bc\" (UID: \"cc6e9b92-6af6-43b2-b88a-9565806323bc\") " Nov 01 23:11:13.359201 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:13.359084 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc6e9b92-6af6-43b2-b88a-9565806323bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc6e9b92-6af6-43b2-b88a-9565806323bc-plugins-config-volume\") pod \"cc6e9b92-6af6-43b2-b88a-9565806323bc\" (UID: \"cc6e9b92-6af6-43b2-b88a-9565806323bc\") " Nov 01 23:11:13.359480 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:13.359444 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc6e9b92-6af6-43b2-b88a-9565806323bc-plugins-config-volume" pod "cc6e9b92-6af6-43b2-b88a-9565806323bc" (UID: "cc6e9b92-6af6-43b2-b88a-9565806323bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:13.359721 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:13.359694 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc6e9b92-6af6-43b2-b88a-9565806323bc-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:13.361276 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:13.361252 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6e9b92-6af6-43b2-b88a-9565806323bc-kube-api-access-bbpt4" pod "cc6e9b92-6af6-43b2-b88a-9565806323bc" (UID: "cc6e9b92-6af6-43b2-b88a-9565806323bc"). InnerVolumeSpecName "kube-api-access-bbpt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:13.461269 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:13.461223 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbpt4\" (UniqueName: \"kubernetes.io/projected/cc6e9b92-6af6-43b2-b88a-9565806323bc-kube-api-access-bbpt4\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:14.020633 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:14.020551 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c6cec6-5fb65c74cf-gzv7m" Nov 01 23:11:14.021174 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:14.020553 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-c6cec6-5fb65c74cf-gzv7m" event={"ID":"cc6e9b92-6af6-43b2-b88a-9565806323bc","Type":"ContainerDied","Data":"91eaaf07c9e842d54cf67dd747fdf5a067e7031f580f9b9f5f2a2cfe6a260e47"} Nov 01 23:11:14.021174 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:14.020696 5011 scope.go:122] "RemoveContainer" containerID="5c501b0add172d2f58632d62fe5d1f4d05b807075dea687e75afa49bfc800480" Nov 01 23:11:14.022252 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:14.022223 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c6cec6-5dbc7cdf79-fv8g6" event={"ID":"39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7","Type":"ContainerDied","Data":"3787ed537d661e018e342c4dfd9c1919cd67f454b44e87af2b8a80ac680090e1"} Nov 01 23:11:14.022407 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:14.022322 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c6cec6-5dbc7cdf79-fv8g6" Nov 01 23:11:14.031081 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:14.031059 5011 scope.go:122] "RemoveContainer" containerID="67b23cc879b660a9eb675764058dac8404c53bd19d65f6f36bbe6f35e62452f2" Nov 01 23:11:14.048262 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:14.048221 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-c6cec6-5fb65c74cf-gzv7m"] Nov 01 23:11:14.052471 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:14.052444 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-c6cec6-5fb65c74cf-gzv7m"] Nov 01 23:11:14.062875 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:14.062793 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c6cec6-5dbc7cdf79-fv8g6"] Nov 01 23:11:14.066294 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:14.066256 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-c6cec6-5dbc7cdf79-fv8g6"] Nov 01 23:11:15.354094 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:15.354062 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:11:15.486596 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:15.486561 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d295266d-5d50-4aae-a0ee-d86513a36f34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d295266d-5d50-4aae-a0ee-d86513a36f34-istiod-ca-cert\") pod \"d295266d-5d50-4aae-a0ee-d86513a36f34\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " Nov 01 23:11:15.486812 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:15.486608 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-credential-socket\") pod \"d295266d-5d50-4aae-a0ee-d86513a36f34\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " Nov 01 23:11:15.486812 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:15.486656 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-workload-socket\") pod \"d295266d-5d50-4aae-a0ee-d86513a36f34\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " Nov 01 23:11:15.486812 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:15.486770 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-workload-certs\") pod \"d295266d-5d50-4aae-a0ee-d86513a36f34\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " Nov 01 23:11:15.487045 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:15.486878 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-istio-envoy\") pod \"d295266d-5d50-4aae-a0ee-d86513a36f34\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " Nov 01 23:11:15.487045 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:15.486978 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-istio-data\") pod \"d295266d-5d50-4aae-a0ee-d86513a36f34\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " Nov 01 23:11:15.487045 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:15.486987 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d295266d-5d50-4aae-a0ee-d86513a36f34-istiod-ca-cert" pod "d295266d-5d50-4aae-a0ee-d86513a36f34" (UID: "d295266d-5d50-4aae-a0ee-d86513a36f34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:15.487045 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:15.487012 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d295266d-5d50-4aae-a0ee-d86513a36f34-kube-api-access-2sqr8\" (UniqueName: \"kubernetes.io/projected/d295266d-5d50-4aae-a0ee-d86513a36f34-kube-api-access-2sqr8\") pod \"d295266d-5d50-4aae-a0ee-d86513a36f34\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " Nov 01 23:11:15.487330 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:15.487053 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d295266d-5d50-4aae-a0ee-d86513a36f34-istio-token\" (UniqueName: \"kubernetes.io/projected/d295266d-5d50-4aae-a0ee-d86513a36f34-istio-token\") pod \"d295266d-5d50-4aae-a0ee-d86513a36f34\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " Nov 01 23:11:15.487330 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:15.487126 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d295266d-5d50-4aae-a0ee-d86513a36f34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d295266d-5d50-4aae-a0ee-d86513a36f34-istio-podinfo\") pod \"d295266d-5d50-4aae-a0ee-d86513a36f34\" (UID: \"d295266d-5d50-4aae-a0ee-d86513a36f34\") " Nov 01 23:11:15.487330 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:15.487257 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-workload-certs" pod "d295266d-5d50-4aae-a0ee-d86513a36f34" (UID: "d295266d-5d50-4aae-a0ee-d86513a36f34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:15.487501 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:15.487414 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-credential-socket" pod "d295266d-5d50-4aae-a0ee-d86513a36f34" (UID: "d295266d-5d50-4aae-a0ee-d86513a36f34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:15.488155 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:15.487729 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:15.488155 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:15.487789 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d295266d-5d50-4aae-a0ee-d86513a36f34-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:15.488155 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:15.487807 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:15.488155 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:15.487881 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-istio-data" pod "d295266d-5d50-4aae-a0ee-d86513a36f34" (UID: "d295266d-5d50-4aae-a0ee-d86513a36f34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:15.488155 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:15.487905 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-workload-socket" pod "d295266d-5d50-4aae-a0ee-d86513a36f34" (UID: "d295266d-5d50-4aae-a0ee-d86513a36f34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:15.490981 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:15.490948 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-istio-envoy" pod "d295266d-5d50-4aae-a0ee-d86513a36f34" (UID: "d295266d-5d50-4aae-a0ee-d86513a36f34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:15.491092 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:15.491056 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d295266d-5d50-4aae-a0ee-d86513a36f34-kube-api-access-2sqr8" pod "d295266d-5d50-4aae-a0ee-d86513a36f34" (UID: "d295266d-5d50-4aae-a0ee-d86513a36f34"). InnerVolumeSpecName "kube-api-access-2sqr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:15.491808 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:15.491782 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d295266d-5d50-4aae-a0ee-d86513a36f34-istio-podinfo" pod "d295266d-5d50-4aae-a0ee-d86513a36f34" (UID: "d295266d-5d50-4aae-a0ee-d86513a36f34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:11:15.491948 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:15.491925 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d295266d-5d50-4aae-a0ee-d86513a36f34-istio-token" pod "d295266d-5d50-4aae-a0ee-d86513a36f34" (UID: "d295266d-5d50-4aae-a0ee-d86513a36f34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:15.589718 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:15.589625 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:15.589718 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:15.589660 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:15.589718 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:15.589675 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sqr8\" (UniqueName: \"kubernetes.io/projected/d295266d-5d50-4aae-a0ee-d86513a36f34-kube-api-access-2sqr8\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:15.589718 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:15.589689 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d295266d-5d50-4aae-a0ee-d86513a36f34-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:15.589718 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:15.589701 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d295266d-5d50-4aae-a0ee-d86513a36f34-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:15.589718 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:15.589714 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d295266d-5d50-4aae-a0ee-d86513a36f34-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:15.749118 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:15.749082 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7" path="/var/lib/kubelet/pods/39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7/volumes" Nov 01 23:11:15.749624 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:15.749602 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc6e9b92-6af6-43b2-b88a-9565806323bc" path="/var/lib/kubelet/pods/cc6e9b92-6af6-43b2-b88a-9565806323bc/volumes" Nov 01 23:11:16.038506 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:16.038466 5011 generic.go:356] "Generic (PLEG): container finished" podID="d295266d-5d50-4aae-a0ee-d86513a36f34" containerID="f545ecb3a6ae9c637aa4de7689c261e1f5b9833cd6d45d0a6ebf21c12ad15b8d" exitCode=0 Nov 01 23:11:16.038693 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:16.038526 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" Nov 01 23:11:16.038693 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:16.038559 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" event={"ID":"d295266d-5d50-4aae-a0ee-d86513a36f34","Type":"ContainerDied","Data":"f545ecb3a6ae9c637aa4de7689c261e1f5b9833cd6d45d0a6ebf21c12ad15b8d"} Nov 01 23:11:16.038693 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:16.038612 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245" event={"ID":"d295266d-5d50-4aae-a0ee-d86513a36f34","Type":"ContainerDied","Data":"df92e351f2882cf1b28396c8bb8120d51a9ac9d4113b0d8e8dee134088a28264"} Nov 01 23:11:16.038693 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:16.038634 5011 scope.go:122] "RemoveContainer" containerID="f545ecb3a6ae9c637aa4de7689c261e1f5b9833cd6d45d0a6ebf21c12ad15b8d" Nov 01 23:11:16.048907 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:16.048888 5011 scope.go:122] "RemoveContainer" containerID="88d917aa5db9d3054d9536a28d70c0834b7e5c53af461700d1c6f3fcce812380" Nov 01 23:11:16.057725 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:16.057701 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245"] Nov 01 23:11:16.058855 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:16.058817 5011 scope.go:122] "RemoveContainer" containerID="f545ecb3a6ae9c637aa4de7689c261e1f5b9833cd6d45d0a6ebf21c12ad15b8d" Nov 01 23:11:16.059173 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:11:16.059153 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f545ecb3a6ae9c637aa4de7689c261e1f5b9833cd6d45d0a6ebf21c12ad15b8d\": container with ID starting with f545ecb3a6ae9c637aa4de7689c261e1f5b9833cd6d45d0a6ebf21c12ad15b8d not found: ID does not exist" containerID="f545ecb3a6ae9c637aa4de7689c261e1f5b9833cd6d45d0a6ebf21c12ad15b8d" Nov 01 23:11:16.059222 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:16.059184 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f545ecb3a6ae9c637aa4de7689c261e1f5b9833cd6d45d0a6ebf21c12ad15b8d"} err="failed to get container status \"f545ecb3a6ae9c637aa4de7689c261e1f5b9833cd6d45d0a6ebf21c12ad15b8d\": rpc error: code = NotFound desc = could not find container \"f545ecb3a6ae9c637aa4de7689c261e1f5b9833cd6d45d0a6ebf21c12ad15b8d\": container with ID starting with f545ecb3a6ae9c637aa4de7689c261e1f5b9833cd6d45d0a6ebf21c12ad15b8d not found: ID does not exist" Nov 01 23:11:16.059222 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:16.059205 5011 scope.go:122] "RemoveContainer" containerID="88d917aa5db9d3054d9536a28d70c0834b7e5c53af461700d1c6f3fcce812380" Nov 01 23:11:16.059457 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:11:16.059438 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88d917aa5db9d3054d9536a28d70c0834b7e5c53af461700d1c6f3fcce812380\": container with ID starting with 88d917aa5db9d3054d9536a28d70c0834b7e5c53af461700d1c6f3fcce812380 not found: ID does not exist" containerID="88d917aa5db9d3054d9536a28d70c0834b7e5c53af461700d1c6f3fcce812380" Nov 01 23:11:16.059517 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:16.059465 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88d917aa5db9d3054d9536a28d70c0834b7e5c53af461700d1c6f3fcce812380"} err="failed to get container status \"88d917aa5db9d3054d9536a28d70c0834b7e5c53af461700d1c6f3fcce812380\": rpc error: code = NotFound desc = could not find container \"88d917aa5db9d3054d9536a28d70c0834b7e5c53af461700d1c6f3fcce812380\": container with ID starting with 88d917aa5db9d3054d9536a28d70c0834b7e5c53af461700d1c6f3fcce812380 not found: ID does not exist" Nov 01 23:11:16.066393 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:16.066371 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-46ce02-openshift-default-67b8c48fdf-pm245"] Nov 01 23:11:16.994863 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:11:16.991899 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:11:17.254378 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:17.251403 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-947846-578fb595bf-l7kpp"] Nov 01 23:11:17.268951 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:17.268922 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-947846-7965f5db47-fr9tf"] Nov 01 23:11:17.747359 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:17.747315 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d295266d-5d50-4aae-a0ee-d86513a36f34" path="/var/lib/kubelet/pods/d295266d-5d50-4aae-a0ee-d86513a36f34/volumes" Nov 01 23:11:20.592054 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:20.592019 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a9f94e-57fdf95d6c-szfm7"] Nov 01 23:11:20.736168 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:20.736131 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c"] Nov 01 23:11:20.736414 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:20.736391 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c" podUID="9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9" containerName="maas-api" containerID="cri-o://c4d79389d01b11e7409cac776c78eb0da8ce192a5782f9d2368f0586e78cdfe4" gracePeriod=30 Nov 01 23:11:21.066716 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:21.066686 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c" Nov 01 23:11:21.067613 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:21.067579 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9-metrics-certs\") pod \"9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9\" (UID: \"9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9\") " Nov 01 23:11:21.068140 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:21.068116 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9-kube-api-access-hkrx4\" (UniqueName: \"kubernetes.io/projected/9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9-kube-api-access-hkrx4\") pod \"9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9\" (UID: \"9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9\") " Nov 01 23:11:21.068398 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:21.068379 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9-openshift-service-ca\") pod \"9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9\" (UID: \"9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9\") " Nov 01 23:11:21.068606 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:21.068589 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9-maas-api-tls\") pod \"9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9\" (UID: \"9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9\") " Nov 01 23:11:21.069140 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:21.069072 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9-openshift-service-ca" pod "9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9" (UID: "9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:21.070122 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:21.070100 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9-openshift-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:21.073411 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:21.073358 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9-kube-api-access-hkrx4" pod "9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9" (UID: "9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9"). InnerVolumeSpecName "kube-api-access-hkrx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:21.073411 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:21.073387 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9-maas-api-tls" pod "9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9" (UID: "9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:21.073646 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:21.073623 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9-metrics-certs" pod "9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9" (UID: "9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:21.076526 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:21.076496 5011 generic.go:356] "Generic (PLEG): container finished" podID="9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9" containerID="c4d79389d01b11e7409cac776c78eb0da8ce192a5782f9d2368f0586e78cdfe4" exitCode=0 Nov 01 23:11:21.076667 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:21.076549 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c" event={"ID":"9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9","Type":"ContainerDied","Data":"c4d79389d01b11e7409cac776c78eb0da8ce192a5782f9d2368f0586e78cdfe4"} Nov 01 23:11:21.076667 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:21.076600 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c" event={"ID":"9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9","Type":"ContainerDied","Data":"aa6cd3667a4498f3fd371107be0965f8fd83606f5f34621f3c841f96dba759fa"} Nov 01 23:11:21.076667 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:21.076630 5011 scope.go:122] "RemoveContainer" containerID="c4d79389d01b11e7409cac776c78eb0da8ce192a5782f9d2368f0586e78cdfe4" Nov 01 23:11:21.076882 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:21.076762 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c" Nov 01 23:11:21.107141 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:21.107119 5011 scope.go:122] "RemoveContainer" containerID="c4d79389d01b11e7409cac776c78eb0da8ce192a5782f9d2368f0586e78cdfe4" Nov 01 23:11:21.107567 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:11:21.107541 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4d79389d01b11e7409cac776c78eb0da8ce192a5782f9d2368f0586e78cdfe4\": container with ID starting with c4d79389d01b11e7409cac776c78eb0da8ce192a5782f9d2368f0586e78cdfe4 not found: ID does not exist" containerID="c4d79389d01b11e7409cac776c78eb0da8ce192a5782f9d2368f0586e78cdfe4" Nov 01 23:11:21.107666 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:21.107578 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4d79389d01b11e7409cac776c78eb0da8ce192a5782f9d2368f0586e78cdfe4"} err="failed to get container status \"c4d79389d01b11e7409cac776c78eb0da8ce192a5782f9d2368f0586e78cdfe4\": rpc error: code = NotFound desc = could not find container \"c4d79389d01b11e7409cac776c78eb0da8ce192a5782f9d2368f0586e78cdfe4\": container with ID starting with c4d79389d01b11e7409cac776c78eb0da8ce192a5782f9d2368f0586e78cdfe4 not found: ID does not exist" Nov 01 23:11:21.119017 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:21.118968 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c"] Nov 01 23:11:21.124004 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:21.123978 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a9f94e-f9677d955-gj94c"] Nov 01 23:11:21.171129 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:21.171098 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9-metrics-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:21.171129 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:21.171123 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkrx4\" (UniqueName: \"kubernetes.io/projected/9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9-kube-api-access-hkrx4\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:21.171129 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:21.171135 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9-maas-api-tls\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:21.654224 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:21.654186 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z"] Nov 01 23:11:21.654881 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:21.654474 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z" podUID="f94602ac-6d0b-4d6e-a4e6-dd99179ec535" containerName="maas-api" containerID="cri-o://bfe614735ede9d9a4fb23b29c2b69251d2f3618f68e6c0c81e1cdf7ec9d30eb4" gracePeriod=30 Nov 01 23:11:21.753109 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:21.753068 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9" path="/var/lib/kubelet/pods/9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9/volumes" Nov 01 23:11:21.923429 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:21.923402 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z" Nov 01 23:11:21.989855 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:21.989358 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f94602ac-6d0b-4d6e-a4e6-dd99179ec535-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f94602ac-6d0b-4d6e-a4e6-dd99179ec535-metrics-certs\") pod \"f94602ac-6d0b-4d6e-a4e6-dd99179ec535\" (UID: \"f94602ac-6d0b-4d6e-a4e6-dd99179ec535\") " Nov 01 23:11:21.989855 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:21.989441 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f94602ac-6d0b-4d6e-a4e6-dd99179ec535-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f94602ac-6d0b-4d6e-a4e6-dd99179ec535-maas-api-tls\") pod \"f94602ac-6d0b-4d6e-a4e6-dd99179ec535\" (UID: \"f94602ac-6d0b-4d6e-a4e6-dd99179ec535\") " Nov 01 23:11:21.989855 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:21.989683 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f94602ac-6d0b-4d6e-a4e6-dd99179ec535-kube-api-access-78ns4\" (UniqueName: \"kubernetes.io/projected/f94602ac-6d0b-4d6e-a4e6-dd99179ec535-kube-api-access-78ns4\") pod \"f94602ac-6d0b-4d6e-a4e6-dd99179ec535\" (UID: \"f94602ac-6d0b-4d6e-a4e6-dd99179ec535\") " Nov 01 23:11:21.989855 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:21.989725 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f94602ac-6d0b-4d6e-a4e6-dd99179ec535-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f94602ac-6d0b-4d6e-a4e6-dd99179ec535-openshift-service-ca\") pod \"f94602ac-6d0b-4d6e-a4e6-dd99179ec535\" (UID: \"f94602ac-6d0b-4d6e-a4e6-dd99179ec535\") " Nov 01 23:11:21.990882 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:21.990855 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f94602ac-6d0b-4d6e-a4e6-dd99179ec535-openshift-service-ca" pod "f94602ac-6d0b-4d6e-a4e6-dd99179ec535" (UID: "f94602ac-6d0b-4d6e-a4e6-dd99179ec535"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:22.006519 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:22.006456 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f94602ac-6d0b-4d6e-a4e6-dd99179ec535-kube-api-access-78ns4" pod "f94602ac-6d0b-4d6e-a4e6-dd99179ec535" (UID: "f94602ac-6d0b-4d6e-a4e6-dd99179ec535"). InnerVolumeSpecName "kube-api-access-78ns4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:22.025291 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:22.025248 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f94602ac-6d0b-4d6e-a4e6-dd99179ec535-metrics-certs" pod "f94602ac-6d0b-4d6e-a4e6-dd99179ec535" (UID: "f94602ac-6d0b-4d6e-a4e6-dd99179ec535"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:22.029178 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:22.029121 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f94602ac-6d0b-4d6e-a4e6-dd99179ec535-maas-api-tls" pod "f94602ac-6d0b-4d6e-a4e6-dd99179ec535" (UID: "f94602ac-6d0b-4d6e-a4e6-dd99179ec535"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:22.091144 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:22.091114 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78ns4\" (UniqueName: \"kubernetes.io/projected/f94602ac-6d0b-4d6e-a4e6-dd99179ec535-kube-api-access-78ns4\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:22.091144 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:22.091140 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f94602ac-6d0b-4d6e-a4e6-dd99179ec535-openshift-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:22.091144 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:22.091153 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f94602ac-6d0b-4d6e-a4e6-dd99179ec535-metrics-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:22.091448 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:22.091167 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f94602ac-6d0b-4d6e-a4e6-dd99179ec535-maas-api-tls\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:22.091685 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:22.091657 5011 generic.go:356] "Generic (PLEG): container finished" podID="f94602ac-6d0b-4d6e-a4e6-dd99179ec535" containerID="bfe614735ede9d9a4fb23b29c2b69251d2f3618f68e6c0c81e1cdf7ec9d30eb4" exitCode=0 Nov 01 23:11:22.091798 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:22.091725 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z" Nov 01 23:11:22.091798 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:22.091738 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z" event={"ID":"f94602ac-6d0b-4d6e-a4e6-dd99179ec535","Type":"ContainerDied","Data":"bfe614735ede9d9a4fb23b29c2b69251d2f3618f68e6c0c81e1cdf7ec9d30eb4"} Nov 01 23:11:22.091928 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:22.091801 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z" event={"ID":"f94602ac-6d0b-4d6e-a4e6-dd99179ec535","Type":"ContainerDied","Data":"fc24015b254d9365f8b602ea18f50860875033de6168f768f256def7ed41051c"} Nov 01 23:11:22.091928 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:22.091825 5011 scope.go:122] "RemoveContainer" containerID="bfe614735ede9d9a4fb23b29c2b69251d2f3618f68e6c0c81e1cdf7ec9d30eb4" Nov 01 23:11:22.106055 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:22.106033 5011 scope.go:122] "RemoveContainer" containerID="bfe614735ede9d9a4fb23b29c2b69251d2f3618f68e6c0c81e1cdf7ec9d30eb4" Nov 01 23:11:22.106424 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:11:22.106402 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfe614735ede9d9a4fb23b29c2b69251d2f3618f68e6c0c81e1cdf7ec9d30eb4\": container with ID starting with bfe614735ede9d9a4fb23b29c2b69251d2f3618f68e6c0c81e1cdf7ec9d30eb4 not found: ID does not exist" containerID="bfe614735ede9d9a4fb23b29c2b69251d2f3618f68e6c0c81e1cdf7ec9d30eb4" Nov 01 23:11:22.106524 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:22.106439 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfe614735ede9d9a4fb23b29c2b69251d2f3618f68e6c0c81e1cdf7ec9d30eb4"} err="failed to get container status \"bfe614735ede9d9a4fb23b29c2b69251d2f3618f68e6c0c81e1cdf7ec9d30eb4\": rpc error: code = NotFound desc = could not find container \"bfe614735ede9d9a4fb23b29c2b69251d2f3618f68e6c0c81e1cdf7ec9d30eb4\": container with ID starting with bfe614735ede9d9a4fb23b29c2b69251d2f3618f68e6c0c81e1cdf7ec9d30eb4 not found: ID does not exist" Nov 01 23:11:22.120405 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:22.120351 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z"] Nov 01 23:11:22.123488 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:22.123458 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-947846-54955bd588-nwx9z"] Nov 01 23:11:22.252565 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:22.252466 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-947846-578fb595bf-l7kpp" podUID="9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729" containerName="payload-processing" containerID="cri-o://12a03fdb3b9d6713424de68e36f661cd49058a089f11a547ad861208f4d452d2" gracePeriod=25 Nov 01 23:11:22.270057 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:22.270001 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-947846-7965f5db47-fr9tf" podUID="6bda6fa3-e106-47fb-91d8-1d150a0249a8" containerName="payload-pre-processing" containerID="cri-o://971e5751bf228ceff3e5a624c1e3662a33416bb29eb881e5fea22504ab0462c2" gracePeriod=25 Nov 01 23:11:23.197365 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:23.197330 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-947846-7965f5db47-fr9tf" podUID="6bda6fa3-e106-47fb-91d8-1d150a0249a8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:11:23.746446 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:23.746406 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f94602ac-6d0b-4d6e-a4e6-dd99179ec535" path="/var/lib/kubelet/pods/f94602ac-6d0b-4d6e-a4e6-dd99179ec535/volumes" Nov 01 23:11:25.592796 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:25.592732 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a9f94e-57fdf95d6c-szfm7" podUID="0beecaf1-fcc2-4d74-ba44-005ecf40b355" containerName="payload-processing" containerID="cri-o://b8300abb8b9986e6ec85bb5138fc9bce64590037e39ce93373a6d2622ce66d05" gracePeriod=25 Nov 01 23:11:29.858364 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:29.858329 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574"] Nov 01 23:11:29.859013 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:29.858612 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" podUID="b054e19b-487f-4cde-9752-607550ccb6c7" containerName="istio-proxy" containerID="cri-o://1f4c4cc28fc85669a438aa6efec6ab97dd5d7e5716c03df31fbb355a3d889c08" gracePeriod=30 Nov 01 23:11:32.031528 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:32.031491 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:11:32.186391 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:32.186345 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-947846-578fb595bf-l7kpp" podUID="9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:11:33.197245 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:33.197210 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-947846-7965f5db47-fr9tf" podUID="6bda6fa3-e106-47fb-91d8-1d150a0249a8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:11:35.070019 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.069968 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a9f94e-57fdf95d6c-szfm7" podUID="0beecaf1-fcc2-4d74-ba44-005ecf40b355" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:11:35.128578 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.128507 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 23:11:35.193212 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.193172 5011 generic.go:356] "Generic (PLEG): container finished" podID="b054e19b-487f-4cde-9752-607550ccb6c7" containerID="1f4c4cc28fc85669a438aa6efec6ab97dd5d7e5716c03df31fbb355a3d889c08" exitCode=0 Nov 01 23:11:35.193407 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.193282 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" Nov 01 23:11:35.193407 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.193302 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" event={"ID":"b054e19b-487f-4cde-9752-607550ccb6c7","Type":"ContainerDied","Data":"1f4c4cc28fc85669a438aa6efec6ab97dd5d7e5716c03df31fbb355a3d889c08"} Nov 01 23:11:35.193407 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.193363 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574" event={"ID":"b054e19b-487f-4cde-9752-607550ccb6c7","Type":"ContainerDied","Data":"cfc2e01972998b9da625fe98e83ccb8a099047aebb2a1ccaa9d5091f855e1d9f"} Nov 01 23:11:35.193407 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.193395 5011 scope.go:122] "RemoveContainer" containerID="1f4c4cc28fc85669a438aa6efec6ab97dd5d7e5716c03df31fbb355a3d889c08" Nov 01 23:11:35.205468 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.205441 5011 scope.go:122] "RemoveContainer" containerID="da35b309d0236f9fdb10c5ed70695ef90d376636ba9b32715e055500e7fab362" Nov 01 23:11:35.222791 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.222566 5011 scope.go:122] "RemoveContainer" containerID="1f4c4cc28fc85669a438aa6efec6ab97dd5d7e5716c03df31fbb355a3d889c08" Nov 01 23:11:35.224015 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:11:35.223981 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f4c4cc28fc85669a438aa6efec6ab97dd5d7e5716c03df31fbb355a3d889c08\": container with ID starting with 1f4c4cc28fc85669a438aa6efec6ab97dd5d7e5716c03df31fbb355a3d889c08 not found: ID does not exist" containerID="1f4c4cc28fc85669a438aa6efec6ab97dd5d7e5716c03df31fbb355a3d889c08" Nov 01 23:11:35.224151 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.224025 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f4c4cc28fc85669a438aa6efec6ab97dd5d7e5716c03df31fbb355a3d889c08"} err="failed to get container status \"1f4c4cc28fc85669a438aa6efec6ab97dd5d7e5716c03df31fbb355a3d889c08\": rpc error: code = NotFound desc = could not find container \"1f4c4cc28fc85669a438aa6efec6ab97dd5d7e5716c03df31fbb355a3d889c08\": container with ID starting with 1f4c4cc28fc85669a438aa6efec6ab97dd5d7e5716c03df31fbb355a3d889c08 not found: ID does not exist" Nov 01 23:11:35.224151 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.224052 5011 scope.go:122] "RemoveContainer" containerID="da35b309d0236f9fdb10c5ed70695ef90d376636ba9b32715e055500e7fab362" Nov 01 23:11:35.224437 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:11:35.224416 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da35b309d0236f9fdb10c5ed70695ef90d376636ba9b32715e055500e7fab362\": container with ID starting with da35b309d0236f9fdb10c5ed70695ef90d376636ba9b32715e055500e7fab362 not found: ID does not exist" containerID="da35b309d0236f9fdb10c5ed70695ef90d376636ba9b32715e055500e7fab362" Nov 01 23:11:35.224501 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.224443 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da35b309d0236f9fdb10c5ed70695ef90d376636ba9b32715e055500e7fab362"} err="failed to get container status \"da35b309d0236f9fdb10c5ed70695ef90d376636ba9b32715e055500e7fab362\": rpc error: code = NotFound desc = could not find container \"da35b309d0236f9fdb10c5ed70695ef90d376636ba9b32715e055500e7fab362\": container with ID starting with da35b309d0236f9fdb10c5ed70695ef90d376636ba9b32715e055500e7fab362 not found: ID does not exist" Nov 01 23:11:35.255570 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.255540 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n"] Nov 01 23:11:35.272809 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.272775 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw"] Nov 01 23:11:35.299228 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.299182 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b054e19b-487f-4cde-9752-607550ccb6c7-kube-api-access-v59nf\" (UniqueName: \"kubernetes.io/projected/b054e19b-487f-4cde-9752-607550ccb6c7-kube-api-access-v59nf\") pod \"b054e19b-487f-4cde-9752-607550ccb6c7\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " Nov 01 23:11:35.300194 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.299272 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-credential-socket\") pod \"b054e19b-487f-4cde-9752-607550ccb6c7\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " Nov 01 23:11:35.300194 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.299312 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-istio-envoy\") pod \"b054e19b-487f-4cde-9752-607550ccb6c7\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " Nov 01 23:11:35.300194 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.299339 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b054e19b-487f-4cde-9752-607550ccb6c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b054e19b-487f-4cde-9752-607550ccb6c7-istiod-ca-cert\") pod \"b054e19b-487f-4cde-9752-607550ccb6c7\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " Nov 01 23:11:35.300194 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.299403 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-workload-socket\") pod \"b054e19b-487f-4cde-9752-607550ccb6c7\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " Nov 01 23:11:35.300194 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.299573 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b054e19b-487f-4cde-9752-607550ccb6c7-istio-token\" (UniqueName: \"kubernetes.io/projected/b054e19b-487f-4cde-9752-607550ccb6c7-istio-token\") pod \"b054e19b-487f-4cde-9752-607550ccb6c7\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " Nov 01 23:11:35.300194 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.299658 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b054e19b-487f-4cde-9752-607550ccb6c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b054e19b-487f-4cde-9752-607550ccb6c7-istio-podinfo\") pod \"b054e19b-487f-4cde-9752-607550ccb6c7\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " Nov 01 23:11:35.300194 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.299716 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-workload-certs\") pod \"b054e19b-487f-4cde-9752-607550ccb6c7\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " Nov 01 23:11:35.300194 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.299749 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-istio-data\") pod \"b054e19b-487f-4cde-9752-607550ccb6c7\" (UID: \"b054e19b-487f-4cde-9752-607550ccb6c7\") " Nov 01 23:11:35.300194 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.299932 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-credential-socket" pod "b054e19b-487f-4cde-9752-607550ccb6c7" (UID: "b054e19b-487f-4cde-9752-607550ccb6c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:35.300194 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.299969 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b054e19b-487f-4cde-9752-607550ccb6c7-istiod-ca-cert" pod "b054e19b-487f-4cde-9752-607550ccb6c7" (UID: "b054e19b-487f-4cde-9752-607550ccb6c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:35.300726 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.300215 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-workload-socket" pod "b054e19b-487f-4cde-9752-607550ccb6c7" (UID: "b054e19b-487f-4cde-9752-607550ccb6c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:35.300726 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.300425 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-istio-data" pod "b054e19b-487f-4cde-9752-607550ccb6c7" (UID: "b054e19b-487f-4cde-9752-607550ccb6c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:35.300726 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.300594 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-workload-certs" pod "b054e19b-487f-4cde-9752-607550ccb6c7" (UID: "b054e19b-487f-4cde-9752-607550ccb6c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:35.300726 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.300570 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:35.300981 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.300744 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b054e19b-487f-4cde-9752-607550ccb6c7-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:35.300981 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.300761 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:35.302278 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.302247 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b054e19b-487f-4cde-9752-607550ccb6c7-istio-podinfo" pod "b054e19b-487f-4cde-9752-607550ccb6c7" (UID: "b054e19b-487f-4cde-9752-607550ccb6c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:11:35.302468 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.302448 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b054e19b-487f-4cde-9752-607550ccb6c7-kube-api-access-v59nf" pod "b054e19b-487f-4cde-9752-607550ccb6c7" (UID: "b054e19b-487f-4cde-9752-607550ccb6c7"). InnerVolumeSpecName "kube-api-access-v59nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:35.302562 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.302489 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b054e19b-487f-4cde-9752-607550ccb6c7-istio-token" pod "b054e19b-487f-4cde-9752-607550ccb6c7" (UID: "b054e19b-487f-4cde-9752-607550ccb6c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:35.302907 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.302884 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-istio-envoy" pod "b054e19b-487f-4cde-9752-607550ccb6c7" (UID: "b054e19b-487f-4cde-9752-607550ccb6c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:35.406644 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.403176 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b054e19b-487f-4cde-9752-607550ccb6c7-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:35.406644 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.403219 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b054e19b-487f-4cde-9752-607550ccb6c7-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:35.406644 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.403235 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:35.406644 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.403249 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:35.406644 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.403263 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v59nf\" (UniqueName: \"kubernetes.io/projected/b054e19b-487f-4cde-9752-607550ccb6c7-kube-api-access-v59nf\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:35.406644 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.403280 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b054e19b-487f-4cde-9752-607550ccb6c7-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:35.518900 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.518822 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574"] Nov 01 23:11:35.521618 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.521587 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-947846-openshift-default-857dff8b7-ss574"] Nov 01 23:11:35.746721 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:35.746684 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b054e19b-487f-4cde-9752-607550ccb6c7" path="/var/lib/kubelet/pods/b054e19b-487f-4cde-9752-607550ccb6c7/volumes" Nov 01 23:11:36.722486 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:36.722453 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf"] Nov 01 23:11:36.722977 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:36.722679 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" podUID="efff7825-e6d6-4049-a3f2-620a4aeab8a2" containerName="istio-proxy" containerID="cri-o://a8f369b2610076be7250803dcc77123f081a6b0aa2ff9aba8d90509756e2cdf9" gracePeriod=30 Nov 01 23:11:36.730457 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:36.730424 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg"] Nov 01 23:11:36.730696 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:36.730673 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" podUID="80ac6525-b39e-4774-b781-0b07dc22f7ab" containerName="istio-proxy" containerID="cri-o://a7750b1025bec3dbe324e2a07edadd027607b8b9d898af3d17d085f88554cf25" gracePeriod=30 Nov 01 23:11:37.863433 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.863401 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd"] Nov 01 23:11:37.863999 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.863983 5011 cpu_manager.go:414] "containerMap: removing container" podUID="f94602ac-6d0b-4d6e-a4e6-dd99179ec535" containerName="maas-api" Nov 01 23:11:37.863999 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.863998 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f94602ac-6d0b-4d6e-a4e6-dd99179ec535" containerName="maas-api" Nov 01 23:11:37.864155 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864011 5011 cpu_manager.go:414] "containerMap: removing container" podUID="9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9" containerName="maas-api" Nov 01 23:11:37.864155 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864016 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9" containerName="maas-api" Nov 01 23:11:37.864155 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864027 5011 cpu_manager.go:414] "containerMap: removing container" podUID="39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7" containerName="payload-pre-processing" Nov 01 23:11:37.864155 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864032 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7" containerName="payload-pre-processing" Nov 01 23:11:37.864155 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864044 5011 cpu_manager.go:414] "containerMap: removing container" podUID="e3a83ece-8434-49fc-a01f-2f926936384d" containerName="payload-processing" Nov 01 23:11:37.864155 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864049 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3a83ece-8434-49fc-a01f-2f926936384d" containerName="payload-processing" Nov 01 23:11:37.864155 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864055 5011 cpu_manager.go:414] "containerMap: removing container" podUID="d295266d-5d50-4aae-a0ee-d86513a36f34" containerName="istio-proxy" Nov 01 23:11:37.864155 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864059 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d295266d-5d50-4aae-a0ee-d86513a36f34" containerName="istio-proxy" Nov 01 23:11:37.864155 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864081 5011 cpu_manager.go:414] "containerMap: removing container" podUID="b054e19b-487f-4cde-9752-607550ccb6c7" containerName="istio-proxy" Nov 01 23:11:37.864155 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864086 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b054e19b-487f-4cde-9752-607550ccb6c7" containerName="istio-proxy" Nov 01 23:11:37.864155 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864092 5011 cpu_manager.go:414] "containerMap: removing container" podUID="b054e19b-487f-4cde-9752-607550ccb6c7" containerName="istio-proxy" Nov 01 23:11:37.864155 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864096 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b054e19b-487f-4cde-9752-607550ccb6c7" containerName="istio-proxy" Nov 01 23:11:37.864155 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864105 5011 cpu_manager.go:414] "containerMap: removing container" podUID="cc6e9b92-6af6-43b2-b88a-9565806323bc" containerName="payload-processing" Nov 01 23:11:37.864155 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864110 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc6e9b92-6af6-43b2-b88a-9565806323bc" containerName="payload-processing" Nov 01 23:11:37.864155 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864118 5011 cpu_manager.go:414] "containerMap: removing container" podUID="d295266d-5d50-4aae-a0ee-d86513a36f34" containerName="istio-proxy" Nov 01 23:11:37.864155 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864122 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d295266d-5d50-4aae-a0ee-d86513a36f34" containerName="istio-proxy" Nov 01 23:11:37.864155 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864130 5011 cpu_manager.go:414] "containerMap: removing container" podUID="07a622eb-374c-400c-bb23-daccb27e6a3a" containerName="istio-proxy" Nov 01 23:11:37.864155 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864134 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a622eb-374c-400c-bb23-daccb27e6a3a" containerName="istio-proxy" Nov 01 23:11:37.864155 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864145 5011 cpu_manager.go:414] "containerMap: removing container" podUID="ed165c04-d90b-45c4-8022-670fb4f2b61c" containerName="istio-proxy" Nov 01 23:11:37.864155 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864150 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed165c04-d90b-45c4-8022-670fb4f2b61c" containerName="istio-proxy" Nov 01 23:11:37.864155 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864158 5011 cpu_manager.go:414] "containerMap: removing container" podUID="be85990a-36c7-4d36-8867-28b4c83867d8" containerName="istio-proxy" Nov 01 23:11:37.864155 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864162 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be85990a-36c7-4d36-8867-28b4c83867d8" containerName="istio-proxy" Nov 01 23:11:37.865364 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864236 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="d295266d-5d50-4aae-a0ee-d86513a36f34" containerName="istio-proxy" Nov 01 23:11:37.865364 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864245 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7b301bf-5694-40b2-8789-07555de1ed0f" containerName="istio-proxy" Nov 01 23:11:37.865364 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864254 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed165c04-d90b-45c4-8022-670fb4f2b61c" containerName="istio-proxy" Nov 01 23:11:37.865364 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864262 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="f94602ac-6d0b-4d6e-a4e6-dd99179ec535" containerName="maas-api" Nov 01 23:11:37.865364 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864267 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c0fb1a7-6aab-4fb7-9352-ed1d6272b1e9" containerName="maas-api" Nov 01 23:11:37.865364 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864273 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="39cc4cd6-52b3-4ee8-8fa0-8018905ddcd7" containerName="payload-pre-processing" Nov 01 23:11:37.865364 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864278 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3a83ece-8434-49fc-a01f-2f926936384d" containerName="payload-processing" Nov 01 23:11:37.865364 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864284 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="d295266d-5d50-4aae-a0ee-d86513a36f34" containerName="istio-proxy" Nov 01 23:11:37.865364 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864292 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="b054e19b-487f-4cde-9752-607550ccb6c7" containerName="istio-proxy" Nov 01 23:11:37.865364 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864298 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="b054e19b-487f-4cde-9752-607550ccb6c7" containerName="istio-proxy" Nov 01 23:11:37.865364 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.864304 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc6e9b92-6af6-43b2-b88a-9565806323bc" containerName="payload-processing" Nov 01 23:11:37.869222 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.869198 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:37.878266 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.878212 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-dc635e-openshift-default-dockercfg-fjhdk\"" Nov 01 23:11:37.878902 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.878871 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9643e76a-dfa8-42b1-8270-cdd219ceceac-istio-token\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:37.879014 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.878969 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-workload-socket\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:37.879088 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.879012 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-istio-data\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:37.879148 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.879088 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-credential-socket\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:37.879148 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.879119 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-istio-envoy\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:37.879257 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.879190 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdxdw\" (UniqueName: \"kubernetes.io/projected/9643e76a-dfa8-42b1-8270-cdd219ceceac-kube-api-access-xdxdw\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:37.879257 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.879222 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-workload-certs\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:37.879370 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.879302 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9643e76a-dfa8-42b1-8270-cdd219ceceac-istiod-ca-cert\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:37.879370 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.879328 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9643e76a-dfa8-42b1-8270-cdd219ceceac-istio-podinfo\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:37.882534 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.882273 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd"] Nov 01 23:11:37.980946 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.980909 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9643e76a-dfa8-42b1-8270-cdd219ceceac-istiod-ca-cert\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:37.980946 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.980949 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9643e76a-dfa8-42b1-8270-cdd219ceceac-istio-podinfo\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:37.981212 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.980999 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9643e76a-dfa8-42b1-8270-cdd219ceceac-istio-token\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:37.981212 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.981038 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-workload-socket\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:37.981212 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.981069 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-istio-data\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:37.981212 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.981132 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-credential-socket\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:37.981212 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.981160 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-istio-envoy\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:37.981212 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.981211 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdxdw\" (UniqueName: \"kubernetes.io/projected/9643e76a-dfa8-42b1-8270-cdd219ceceac-kube-api-access-xdxdw\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:37.981572 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.981238 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-workload-certs\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:37.981572 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.981484 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-workload-socket\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:37.981688 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.981645 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-workload-certs\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:37.981688 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.981652 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-istio-data\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:37.981791 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.981690 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-credential-socket\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:37.981791 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.981752 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9643e76a-dfa8-42b1-8270-cdd219ceceac-istiod-ca-cert\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:37.983644 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.983621 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-istio-envoy\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:37.983993 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.983969 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9643e76a-dfa8-42b1-8270-cdd219ceceac-istio-podinfo\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:37.989136 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.989113 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9643e76a-dfa8-42b1-8270-cdd219ceceac-istio-token\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:37.989343 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:37.989319 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdxdw\" (UniqueName: \"kubernetes.io/projected/9643e76a-dfa8-42b1-8270-cdd219ceceac-kube-api-access-xdxdw\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:38.186290 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:38.186253 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:38.484187 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:38.484151 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd"] Nov 01 23:11:38.484345 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:11:38.484317 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9643e76a_dfa8_42b1_8270_cdd219ceceac.slice/crio-1a85fb7a48ef01726a33785e2d22cbc856658d951077caf58245d36571dcdb3c WatchSource:0}: Error finding container 1a85fb7a48ef01726a33785e2d22cbc856658d951077caf58245d36571dcdb3c: Status 404 returned error can't find the container with id 1a85fb7a48ef01726a33785e2d22cbc856658d951077caf58245d36571dcdb3c Nov 01 23:11:38.487069 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:38.487030 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:11:38.487162 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:38.487120 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:11:38.487204 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:38.487168 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:11:39.294072 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:39.294037 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" event={"ID":"9643e76a-dfa8-42b1-8270-cdd219ceceac","Type":"ContainerStarted","Data":"e8fcd90bc8481db73938c7b09dbce57de9b21a0f2cd849e8dbe31bb96482e0d9"} Nov 01 23:11:39.294464 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:39.294093 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" event={"ID":"9643e76a-dfa8-42b1-8270-cdd219ceceac","Type":"ContainerStarted","Data":"1a85fb7a48ef01726a33785e2d22cbc856658d951077caf58245d36571dcdb3c"} Nov 01 23:11:39.321239 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:39.320624 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" podStartSLOduration=2.32060436 podStartE2EDuration="2.32060436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:11:39.314259758 +0000 UTC m=+2867.718850403" watchObservedRunningTime="2026-11-01 23:11:39.32060436 +0000 UTC m=+2867.725194993" Nov 01 23:11:40.187406 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:40.187281 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:40.188929 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:40.188902 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 01 23:11:40.189127 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:40.189106 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" podUID="9643e76a-dfa8-42b1-8270-cdd219ceceac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 01 23:11:41.187412 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:41.187375 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 01 23:11:41.187891 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:41.187446 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" podUID="9643e76a-dfa8-42b1-8270-cdd219ceceac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 01 23:11:42.171124 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.171098 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:11:42.174359 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.174334 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:11:42.186592 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.186553 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-947846-578fb595bf-l7kpp" podUID="9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:11:42.186731 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.186566 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 01 23:11:42.186731 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.186676 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" podUID="9643e76a-dfa8-42b1-8270-cdd219ceceac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 01 23:11:42.251859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.242121 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-workload-socket\") pod \"80ac6525-b39e-4774-b781-0b07dc22f7ab\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " Nov 01 23:11:42.251859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.242169 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-istio-envoy\") pod \"80ac6525-b39e-4774-b781-0b07dc22f7ab\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " Nov 01 23:11:42.251859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.242225 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-workload-certs\") pod \"80ac6525-b39e-4774-b781-0b07dc22f7ab\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " Nov 01 23:11:42.251859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.242259 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80ac6525-b39e-4774-b781-0b07dc22f7ab-istio-token\" (UniqueName: \"kubernetes.io/projected/80ac6525-b39e-4774-b781-0b07dc22f7ab-istio-token\") pod \"80ac6525-b39e-4774-b781-0b07dc22f7ab\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " Nov 01 23:11:42.251859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.242313 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-workload-socket\") pod \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " Nov 01 23:11:42.251859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.242357 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efff7825-e6d6-4049-a3f2-620a4aeab8a2-kube-api-access-5q6v6\" (UniqueName: \"kubernetes.io/projected/efff7825-e6d6-4049-a3f2-620a4aeab8a2-kube-api-access-5q6v6\") pod \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " Nov 01 23:11:42.251859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.242392 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istio-podinfo\") pod \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " Nov 01 23:11:42.251859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.242419 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istio-data\") pod \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " Nov 01 23:11:42.251859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.242471 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-credential-socket\") pod \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " Nov 01 23:11:42.251859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.242510 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-credential-socket\") pod \"80ac6525-b39e-4774-b781-0b07dc22f7ab\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " Nov 01 23:11:42.251859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.242571 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80ac6525-b39e-4774-b781-0b07dc22f7ab-kube-api-access-z6h5b\" (UniqueName: \"kubernetes.io/projected/80ac6525-b39e-4774-b781-0b07dc22f7ab-kube-api-access-z6h5b\") pod \"80ac6525-b39e-4774-b781-0b07dc22f7ab\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " Nov 01 23:11:42.251859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.242629 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80ac6525-b39e-4774-b781-0b07dc22f7ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80ac6525-b39e-4774-b781-0b07dc22f7ab-istio-podinfo\") pod \"80ac6525-b39e-4774-b781-0b07dc22f7ab\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " Nov 01 23:11:42.251859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.242675 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-istio-data\") pod \"80ac6525-b39e-4774-b781-0b07dc22f7ab\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " Nov 01 23:11:42.251859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.242709 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80ac6525-b39e-4774-b781-0b07dc22f7ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80ac6525-b39e-4774-b781-0b07dc22f7ab-istiod-ca-cert\") pod \"80ac6525-b39e-4774-b781-0b07dc22f7ab\" (UID: \"80ac6525-b39e-4774-b781-0b07dc22f7ab\") " Nov 01 23:11:42.251859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.242783 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istio-token\" (UniqueName: \"kubernetes.io/projected/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istio-token\") pod \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " Nov 01 23:11:42.251859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.242912 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istio-envoy\") pod \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " Nov 01 23:11:42.251859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.242962 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istiod-ca-cert\") pod \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " Nov 01 23:11:42.251859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.242999 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-workload-certs\") pod \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\" (UID: \"efff7825-e6d6-4049-a3f2-620a4aeab8a2\") " Nov 01 23:11:42.267974 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.264196 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-credential-socket" pod "efff7825-e6d6-4049-a3f2-620a4aeab8a2" (UID: "efff7825-e6d6-4049-a3f2-620a4aeab8a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.267974 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.265123 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-istio-data" pod "80ac6525-b39e-4774-b781-0b07dc22f7ab" (UID: "80ac6525-b39e-4774-b781-0b07dc22f7ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.267974 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.266903 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-credential-socket" pod "80ac6525-b39e-4774-b781-0b07dc22f7ab" (UID: "80ac6525-b39e-4774-b781-0b07dc22f7ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.267974 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.266941 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-workload-socket" pod "80ac6525-b39e-4774-b781-0b07dc22f7ab" (UID: "80ac6525-b39e-4774-b781-0b07dc22f7ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.267974 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.267183 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-workload-socket" pod "efff7825-e6d6-4049-a3f2-620a4aeab8a2" (UID: "efff7825-e6d6-4049-a3f2-620a4aeab8a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.267974 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.267347 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-workload-certs" pod "80ac6525-b39e-4774-b781-0b07dc22f7ab" (UID: "80ac6525-b39e-4774-b781-0b07dc22f7ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.268553 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.268201 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ac6525-b39e-4774-b781-0b07dc22f7ab-istiod-ca-cert" pod "80ac6525-b39e-4774-b781-0b07dc22f7ab" (UID: "80ac6525-b39e-4774-b781-0b07dc22f7ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:42.268553 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.268255 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-workload-certs" pod "efff7825-e6d6-4049-a3f2-620a4aeab8a2" (UID: "efff7825-e6d6-4049-a3f2-620a4aeab8a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.272209 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.272177 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istio-data" pod "efff7825-e6d6-4049-a3f2-620a4aeab8a2" (UID: "efff7825-e6d6-4049-a3f2-620a4aeab8a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.272372 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.272207 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istiod-ca-cert" pod "efff7825-e6d6-4049-a3f2-620a4aeab8a2" (UID: "efff7825-e6d6-4049-a3f2-620a4aeab8a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:42.273706 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.273663 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-istio-envoy" pod "80ac6525-b39e-4774-b781-0b07dc22f7ab" (UID: "80ac6525-b39e-4774-b781-0b07dc22f7ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.274575 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.274510 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efff7825-e6d6-4049-a3f2-620a4aeab8a2-kube-api-access-5q6v6" pod "efff7825-e6d6-4049-a3f2-620a4aeab8a2" (UID: "efff7825-e6d6-4049-a3f2-620a4aeab8a2"). InnerVolumeSpecName "kube-api-access-5q6v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:42.274924 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.274901 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80ac6525-b39e-4774-b781-0b07dc22f7ab-istio-podinfo" pod "80ac6525-b39e-4774-b781-0b07dc22f7ab" (UID: "80ac6525-b39e-4774-b781-0b07dc22f7ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:11:42.275471 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.275442 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istio-podinfo" pod "efff7825-e6d6-4049-a3f2-620a4aeab8a2" (UID: "efff7825-e6d6-4049-a3f2-620a4aeab8a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:11:42.275711 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.275688 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istio-envoy" pod "efff7825-e6d6-4049-a3f2-620a4aeab8a2" (UID: "efff7825-e6d6-4049-a3f2-620a4aeab8a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.276052 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.276030 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ac6525-b39e-4774-b781-0b07dc22f7ab-istio-token" pod "80ac6525-b39e-4774-b781-0b07dc22f7ab" (UID: "80ac6525-b39e-4774-b781-0b07dc22f7ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:42.276263 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.276245 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ac6525-b39e-4774-b781-0b07dc22f7ab-kube-api-access-z6h5b" pod "80ac6525-b39e-4774-b781-0b07dc22f7ab" (UID: "80ac6525-b39e-4774-b781-0b07dc22f7ab"). InnerVolumeSpecName "kube-api-access-z6h5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:42.276425 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.276409 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istio-token" pod "efff7825-e6d6-4049-a3f2-620a4aeab8a2" (UID: "efff7825-e6d6-4049-a3f2-620a4aeab8a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:42.315604 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.315570 5011 generic.go:356] "Generic (PLEG): container finished" podID="efff7825-e6d6-4049-a3f2-620a4aeab8a2" containerID="a8f369b2610076be7250803dcc77123f081a6b0aa2ff9aba8d90509756e2cdf9" exitCode=0 Nov 01 23:11:42.315737 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.315626 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" Nov 01 23:11:42.315737 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.315646 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" event={"ID":"efff7825-e6d6-4049-a3f2-620a4aeab8a2","Type":"ContainerDied","Data":"a8f369b2610076be7250803dcc77123f081a6b0aa2ff9aba8d90509756e2cdf9"} Nov 01 23:11:42.315737 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.315703 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf" event={"ID":"efff7825-e6d6-4049-a3f2-620a4aeab8a2","Type":"ContainerDied","Data":"00c2df41fe7c6a8c64e8ae0c1e0f0a4d33202f79ecd462839c27c628790b22a7"} Nov 01 23:11:42.315737 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.315724 5011 scope.go:122] "RemoveContainer" containerID="a8f369b2610076be7250803dcc77123f081a6b0aa2ff9aba8d90509756e2cdf9" Nov 01 23:11:42.317432 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.317408 5011 generic.go:356] "Generic (PLEG): container finished" podID="80ac6525-b39e-4774-b781-0b07dc22f7ab" containerID="a7750b1025bec3dbe324e2a07edadd027607b8b9d898af3d17d085f88554cf25" exitCode=0 Nov 01 23:11:42.317524 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.317447 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" Nov 01 23:11:42.317524 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.317483 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" event={"ID":"80ac6525-b39e-4774-b781-0b07dc22f7ab","Type":"ContainerDied","Data":"a7750b1025bec3dbe324e2a07edadd027607b8b9d898af3d17d085f88554cf25"} Nov 01 23:11:42.317602 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.317525 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg" event={"ID":"80ac6525-b39e-4774-b781-0b07dc22f7ab","Type":"ContainerDied","Data":"1a879747390a79bdcde8579095c6a3369a3fecbc293e935c8413e19d58002896"} Nov 01 23:11:42.368869 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.368522 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6h5b\" (UniqueName: \"kubernetes.io/projected/80ac6525-b39e-4774-b781-0b07dc22f7ab-kube-api-access-z6h5b\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.368869 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.368560 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80ac6525-b39e-4774-b781-0b07dc22f7ab-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.368869 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.368615 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.368869 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.368628 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80ac6525-b39e-4774-b781-0b07dc22f7ab-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.368869 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.368641 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.368869 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.368691 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.368869 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.368706 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.368869 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.368719 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.368869 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.368776 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.368869 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.368791 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.368869 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.368804 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.368869 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.368867 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80ac6525-b39e-4774-b781-0b07dc22f7ab-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.368869 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.368881 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.369750 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.368894 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q6v6\" (UniqueName: \"kubernetes.io/projected/efff7825-e6d6-4049-a3f2-620a4aeab8a2-kube-api-access-5q6v6\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.369750 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.368909 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.369750 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.368936 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.369750 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.368952 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efff7825-e6d6-4049-a3f2-620a4aeab8a2-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.369750 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.368966 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80ac6525-b39e-4774-b781-0b07dc22f7ab-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.391488 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.391444 5011 scope.go:122] "RemoveContainer" containerID="a8f369b2610076be7250803dcc77123f081a6b0aa2ff9aba8d90509756e2cdf9" Nov 01 23:11:42.391971 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:11:42.391822 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8f369b2610076be7250803dcc77123f081a6b0aa2ff9aba8d90509756e2cdf9\": container with ID starting with a8f369b2610076be7250803dcc77123f081a6b0aa2ff9aba8d90509756e2cdf9 not found: ID does not exist" containerID="a8f369b2610076be7250803dcc77123f081a6b0aa2ff9aba8d90509756e2cdf9" Nov 01 23:11:42.391971 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.391884 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8f369b2610076be7250803dcc77123f081a6b0aa2ff9aba8d90509756e2cdf9"} err="failed to get container status \"a8f369b2610076be7250803dcc77123f081a6b0aa2ff9aba8d90509756e2cdf9\": rpc error: code = NotFound desc = could not find container \"a8f369b2610076be7250803dcc77123f081a6b0aa2ff9aba8d90509756e2cdf9\": container with ID starting with a8f369b2610076be7250803dcc77123f081a6b0aa2ff9aba8d90509756e2cdf9 not found: ID does not exist" Nov 01 23:11:42.391971 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.391910 5011 scope.go:122] "RemoveContainer" containerID="a7750b1025bec3dbe324e2a07edadd027607b8b9d898af3d17d085f88554cf25" Nov 01 23:11:42.404324 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.404299 5011 scope.go:122] "RemoveContainer" containerID="a7750b1025bec3dbe324e2a07edadd027607b8b9d898af3d17d085f88554cf25" Nov 01 23:11:42.405289 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.405223 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf"] Nov 01 23:11:42.405638 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:11:42.405387 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7750b1025bec3dbe324e2a07edadd027607b8b9d898af3d17d085f88554cf25\": container with ID starting with a7750b1025bec3dbe324e2a07edadd027607b8b9d898af3d17d085f88554cf25 not found: ID does not exist" containerID="a7750b1025bec3dbe324e2a07edadd027607b8b9d898af3d17d085f88554cf25" Nov 01 23:11:42.405739 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.405683 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7750b1025bec3dbe324e2a07edadd027607b8b9d898af3d17d085f88554cf25"} err="failed to get container status \"a7750b1025bec3dbe324e2a07edadd027607b8b9d898af3d17d085f88554cf25\": rpc error: code = NotFound desc = could not find container \"a7750b1025bec3dbe324e2a07edadd027607b8b9d898af3d17d085f88554cf25\": container with ID starting with a7750b1025bec3dbe324e2a07edadd027607b8b9d898af3d17d085f88554cf25 not found: ID does not exist" Nov 01 23:11:42.409092 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.409050 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-p7xwf"] Nov 01 23:11:42.423245 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.423193 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg"] Nov 01 23:11:42.431944 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.431897 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-dz2mg"] Nov 01 23:11:42.610248 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.610219 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-947846-7965f5db47-fr9tf" Nov 01 23:11:42.677082 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.677051 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bda6fa3-e106-47fb-91d8-1d150a0249a8-kube-api-access-fn9gr\" (UniqueName: \"kubernetes.io/projected/6bda6fa3-e106-47fb-91d8-1d150a0249a8-kube-api-access-fn9gr\") pod \"6bda6fa3-e106-47fb-91d8-1d150a0249a8\" (UID: \"6bda6fa3-e106-47fb-91d8-1d150a0249a8\") " Nov 01 23:11:42.677263 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.677132 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bda6fa3-e106-47fb-91d8-1d150a0249a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bda6fa3-e106-47fb-91d8-1d150a0249a8-plugins-config-volume\") pod \"6bda6fa3-e106-47fb-91d8-1d150a0249a8\" (UID: \"6bda6fa3-e106-47fb-91d8-1d150a0249a8\") " Nov 01 23:11:42.678558 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.678507 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bda6fa3-e106-47fb-91d8-1d150a0249a8-plugins-config-volume" pod "6bda6fa3-e106-47fb-91d8-1d150a0249a8" (UID: "6bda6fa3-e106-47fb-91d8-1d150a0249a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:42.681297 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.681269 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bda6fa3-e106-47fb-91d8-1d150a0249a8-kube-api-access-fn9gr" pod "6bda6fa3-e106-47fb-91d8-1d150a0249a8" (UID: "6bda6fa3-e106-47fb-91d8-1d150a0249a8"). InnerVolumeSpecName "kube-api-access-fn9gr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:42.682790 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.682465 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-947846-578fb595bf-l7kpp" Nov 01 23:11:42.779100 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.779071 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729-plugins-config-volume\") pod \"9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729\" (UID: \"9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729\") " Nov 01 23:11:42.779271 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.779176 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729-kube-api-access-xcdf2\" (UniqueName: \"kubernetes.io/projected/9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729-kube-api-access-xcdf2\") pod \"9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729\" (UID: \"9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729\") " Nov 01 23:11:42.782340 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.782305 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729-plugins-config-volume" pod "9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729" (UID: "9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:42.782682 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.782658 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn9gr\" (UniqueName: \"kubernetes.io/projected/6bda6fa3-e106-47fb-91d8-1d150a0249a8-kube-api-access-fn9gr\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.782737 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.782689 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bda6fa3-e106-47fb-91d8-1d150a0249a8-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.782737 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.782711 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.785873 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.785685 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729-kube-api-access-xcdf2" pod "9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729" (UID: "9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729"). InnerVolumeSpecName "kube-api-access-xcdf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:42.883427 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:42.883376 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcdf2\" (UniqueName: \"kubernetes.io/projected/9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729-kube-api-access-xcdf2\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:43.186898 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:43.186825 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 01 23:11:43.187092 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:43.186921 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" podUID="9643e76a-dfa8-42b1-8270-cdd219ceceac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 01 23:11:43.324694 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:43.324656 5011 generic.go:356] "Generic (PLEG): container finished" podID="9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729" containerID="12a03fdb3b9d6713424de68e36f661cd49058a089f11a547ad861208f4d452d2" exitCode=0 Nov 01 23:11:43.325180 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:43.324724 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-947846-578fb595bf-l7kpp" Nov 01 23:11:43.325180 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:43.324730 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-947846-578fb595bf-l7kpp" event={"ID":"9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729","Type":"ContainerDied","Data":"12a03fdb3b9d6713424de68e36f661cd49058a089f11a547ad861208f4d452d2"} Nov 01 23:11:43.325180 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:43.324820 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-947846-578fb595bf-l7kpp" event={"ID":"9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729","Type":"ContainerDied","Data":"d963d7caac1cc1a3697e5bbd74b4a5bbe74d22a6f2c4adb710f8d511631f0030"} Nov 01 23:11:43.325180 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:43.324864 5011 scope.go:122] "RemoveContainer" containerID="12a03fdb3b9d6713424de68e36f661cd49058a089f11a547ad861208f4d452d2" Nov 01 23:11:43.327270 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:43.327232 5011 generic.go:356] "Generic (PLEG): container finished" podID="6bda6fa3-e106-47fb-91d8-1d150a0249a8" containerID="971e5751bf228ceff3e5a624c1e3662a33416bb29eb881e5fea22504ab0462c2" exitCode=0 Nov 01 23:11:43.327387 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:43.327280 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-947846-7965f5db47-fr9tf" Nov 01 23:11:43.327387 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:43.327300 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-947846-7965f5db47-fr9tf" event={"ID":"6bda6fa3-e106-47fb-91d8-1d150a0249a8","Type":"ContainerDied","Data":"971e5751bf228ceff3e5a624c1e3662a33416bb29eb881e5fea22504ab0462c2"} Nov 01 23:11:43.327387 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:43.327343 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-947846-7965f5db47-fr9tf" event={"ID":"6bda6fa3-e106-47fb-91d8-1d150a0249a8","Type":"ContainerDied","Data":"af4bcb661f2fae43d3eb1e64228b6105fd64bfb8b23b11f715adac9bff7af32c"} Nov 01 23:11:43.335814 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:43.335788 5011 scope.go:122] "RemoveContainer" containerID="12a03fdb3b9d6713424de68e36f661cd49058a089f11a547ad861208f4d452d2" Nov 01 23:11:43.336165 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:11:43.336135 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12a03fdb3b9d6713424de68e36f661cd49058a089f11a547ad861208f4d452d2\": container with ID starting with 12a03fdb3b9d6713424de68e36f661cd49058a089f11a547ad861208f4d452d2 not found: ID does not exist" containerID="12a03fdb3b9d6713424de68e36f661cd49058a089f11a547ad861208f4d452d2" Nov 01 23:11:43.336248 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:43.336178 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12a03fdb3b9d6713424de68e36f661cd49058a089f11a547ad861208f4d452d2"} err="failed to get container status \"12a03fdb3b9d6713424de68e36f661cd49058a089f11a547ad861208f4d452d2\": rpc error: code = NotFound desc = could not find container \"12a03fdb3b9d6713424de68e36f661cd49058a089f11a547ad861208f4d452d2\": container with ID starting with 12a03fdb3b9d6713424de68e36f661cd49058a089f11a547ad861208f4d452d2 not found: ID does not exist" Nov 01 23:11:43.336248 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:43.336203 5011 scope.go:122] "RemoveContainer" containerID="971e5751bf228ceff3e5a624c1e3662a33416bb29eb881e5fea22504ab0462c2" Nov 01 23:11:43.368870 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:43.368752 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-947846-578fb595bf-l7kpp"] Nov 01 23:11:43.372942 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:43.372911 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-947846-578fb595bf-l7kpp"] Nov 01 23:11:43.374993 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:43.374975 5011 scope.go:122] "RemoveContainer" containerID="971e5751bf228ceff3e5a624c1e3662a33416bb29eb881e5fea22504ab0462c2" Nov 01 23:11:43.375311 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:11:43.375286 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"971e5751bf228ceff3e5a624c1e3662a33416bb29eb881e5fea22504ab0462c2\": container with ID starting with 971e5751bf228ceff3e5a624c1e3662a33416bb29eb881e5fea22504ab0462c2 not found: ID does not exist" containerID="971e5751bf228ceff3e5a624c1e3662a33416bb29eb881e5fea22504ab0462c2" Nov 01 23:11:43.375411 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:43.375318 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"971e5751bf228ceff3e5a624c1e3662a33416bb29eb881e5fea22504ab0462c2"} err="failed to get container status \"971e5751bf228ceff3e5a624c1e3662a33416bb29eb881e5fea22504ab0462c2\": rpc error: code = NotFound desc = could not find container \"971e5751bf228ceff3e5a624c1e3662a33416bb29eb881e5fea22504ab0462c2\": container with ID starting with 971e5751bf228ceff3e5a624c1e3662a33416bb29eb881e5fea22504ab0462c2 not found: ID does not exist" Nov 01 23:11:43.390219 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:43.390193 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-947846-7965f5db47-fr9tf"] Nov 01 23:11:43.396344 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:43.396322 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-947846-7965f5db47-fr9tf"] Nov 01 23:11:43.771689 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:43.771644 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bda6fa3-e106-47fb-91d8-1d150a0249a8" path="/var/lib/kubelet/pods/6bda6fa3-e106-47fb-91d8-1d150a0249a8/volumes" Nov 01 23:11:43.772230 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:43.772208 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80ac6525-b39e-4774-b781-0b07dc22f7ab" path="/var/lib/kubelet/pods/80ac6525-b39e-4774-b781-0b07dc22f7ab/volumes" Nov 01 23:11:43.772646 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:43.772626 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729" path="/var/lib/kubelet/pods/9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729/volumes" Nov 01 23:11:43.773456 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:43.773432 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efff7825-e6d6-4049-a3f2-620a4aeab8a2" path="/var/lib/kubelet/pods/efff7825-e6d6-4049-a3f2-620a4aeab8a2/volumes" Nov 01 23:11:44.187579 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:44.187521 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Nov 01 23:11:44.187746 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:44.187597 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" podUID="9643e76a-dfa8-42b1-8270-cdd219ceceac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Nov 01 23:11:45.065452 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:45.065418 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a9f94e-57fdf95d6c-szfm7" podUID="0beecaf1-fcc2-4d74-ba44-005ecf40b355" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:11:45.191385 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:45.191346 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:45.191688 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:45.191667 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:45.192352 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:45.192331 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:11:45.860209 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:45.860141 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a9f94e-57fdf95d6c-szfm7" Nov 01 23:11:45.931771 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:45.931729 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0beecaf1-fcc2-4d74-ba44-005ecf40b355-kube-api-access-2vhbq\" (UniqueName: \"kubernetes.io/projected/0beecaf1-fcc2-4d74-ba44-005ecf40b355-kube-api-access-2vhbq\") pod \"0beecaf1-fcc2-4d74-ba44-005ecf40b355\" (UID: \"0beecaf1-fcc2-4d74-ba44-005ecf40b355\") " Nov 01 23:11:45.932007 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:45.931779 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0beecaf1-fcc2-4d74-ba44-005ecf40b355-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0beecaf1-fcc2-4d74-ba44-005ecf40b355-plugins-config-volume\") pod \"0beecaf1-fcc2-4d74-ba44-005ecf40b355\" (UID: \"0beecaf1-fcc2-4d74-ba44-005ecf40b355\") " Nov 01 23:11:45.932251 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:45.932225 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0beecaf1-fcc2-4d74-ba44-005ecf40b355-plugins-config-volume" pod "0beecaf1-fcc2-4d74-ba44-005ecf40b355" (UID: "0beecaf1-fcc2-4d74-ba44-005ecf40b355"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:45.932382 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:45.932365 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0beecaf1-fcc2-4d74-ba44-005ecf40b355-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:45.934082 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:45.934055 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0beecaf1-fcc2-4d74-ba44-005ecf40b355-kube-api-access-2vhbq" pod "0beecaf1-fcc2-4d74-ba44-005ecf40b355" (UID: "0beecaf1-fcc2-4d74-ba44-005ecf40b355"). InnerVolumeSpecName "kube-api-access-2vhbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:46.033296 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:46.033264 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vhbq\" (UniqueName: \"kubernetes.io/projected/0beecaf1-fcc2-4d74-ba44-005ecf40b355-kube-api-access-2vhbq\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:46.355039 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:46.355003 5011 generic.go:356] "Generic (PLEG): container finished" podID="0beecaf1-fcc2-4d74-ba44-005ecf40b355" containerID="b8300abb8b9986e6ec85bb5138fc9bce64590037e39ce93373a6d2622ce66d05" exitCode=0 Nov 01 23:11:46.355501 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:46.355048 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a9f94e-57fdf95d6c-szfm7" Nov 01 23:11:46.355501 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:46.355075 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a9f94e-57fdf95d6c-szfm7" event={"ID":"0beecaf1-fcc2-4d74-ba44-005ecf40b355","Type":"ContainerDied","Data":"b8300abb8b9986e6ec85bb5138fc9bce64590037e39ce93373a6d2622ce66d05"} Nov 01 23:11:46.355501 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:46.355133 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a9f94e-57fdf95d6c-szfm7" event={"ID":"0beecaf1-fcc2-4d74-ba44-005ecf40b355","Type":"ContainerDied","Data":"28a272dc4b19a88c33220b30659d53a6ff6da236e4fad55e0ec4c45c0bff3d50"} Nov 01 23:11:46.355501 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:46.355158 5011 scope.go:122] "RemoveContainer" containerID="b8300abb8b9986e6ec85bb5138fc9bce64590037e39ce93373a6d2622ce66d05" Nov 01 23:11:46.366639 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:46.366618 5011 scope.go:122] "RemoveContainer" containerID="b8300abb8b9986e6ec85bb5138fc9bce64590037e39ce93373a6d2622ce66d05" Nov 01 23:11:46.367055 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:11:46.367031 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8300abb8b9986e6ec85bb5138fc9bce64590037e39ce93373a6d2622ce66d05\": container with ID starting with b8300abb8b9986e6ec85bb5138fc9bce64590037e39ce93373a6d2622ce66d05 not found: ID does not exist" containerID="b8300abb8b9986e6ec85bb5138fc9bce64590037e39ce93373a6d2622ce66d05" Nov 01 23:11:46.367138 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:46.367064 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8300abb8b9986e6ec85bb5138fc9bce64590037e39ce93373a6d2622ce66d05"} err="failed to get container status \"b8300abb8b9986e6ec85bb5138fc9bce64590037e39ce93373a6d2622ce66d05\": rpc error: code = NotFound desc = could not find container \"b8300abb8b9986e6ec85bb5138fc9bce64590037e39ce93373a6d2622ce66d05\": container with ID starting with b8300abb8b9986e6ec85bb5138fc9bce64590037e39ce93373a6d2622ce66d05 not found: ID does not exist" Nov 01 23:11:46.382883 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:46.382857 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a9f94e-57fdf95d6c-szfm7"] Nov 01 23:11:46.386889 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:46.386825 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a9f94e-57fdf95d6c-szfm7"] Nov 01 23:11:46.958938 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:11:46.958905 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:11:47.330473 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:47.330393 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-075771-66579df747-cbnsf"] Nov 01 23:11:47.353064 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:47.353027 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-075771-66b777b8dc-cphpc"] Nov 01 23:11:47.747068 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:47.747038 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0beecaf1-fcc2-4d74-ba44-005ecf40b355" path="/var/lib/kubelet/pods/0beecaf1-fcc2-4d74-ba44-005ecf40b355/volumes" Nov 01 23:11:47.801466 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:47.801435 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-56b833-d86bfc667-g45bg"] Nov 01 23:11:47.816164 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:47.816132 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56b833-556fbfdf9-snb4s"] Nov 01 23:11:47.978576 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:47.978542 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb"] Nov 01 23:11:47.978812 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:47.978788 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb" podUID="f45f8128-5315-4bd0-98ed-7f0b57309295" containerName="maas-api" containerID="cri-o://689fa041d4286a4209a45cdcf7db2c59d87f58dc47d279177e7865df0c4fe695" gracePeriod=30 Nov 01 23:11:48.313058 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:48.313029 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb" Nov 01 23:11:48.388523 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:48.388488 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f45f8128-5315-4bd0-98ed-7f0b57309295-kube-api-access-5jpv5\" (UniqueName: \"kubernetes.io/projected/f45f8128-5315-4bd0-98ed-7f0b57309295-kube-api-access-5jpv5\") pod \"f45f8128-5315-4bd0-98ed-7f0b57309295\" (UID: \"f45f8128-5315-4bd0-98ed-7f0b57309295\") " Nov 01 23:11:48.388720 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:48.388588 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f45f8128-5315-4bd0-98ed-7f0b57309295-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f45f8128-5315-4bd0-98ed-7f0b57309295-maas-api-tls\") pod \"f45f8128-5315-4bd0-98ed-7f0b57309295\" (UID: \"f45f8128-5315-4bd0-98ed-7f0b57309295\") " Nov 01 23:11:48.388720 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:48.388705 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f45f8128-5315-4bd0-98ed-7f0b57309295-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f45f8128-5315-4bd0-98ed-7f0b57309295-openshift-service-ca\") pod \"f45f8128-5315-4bd0-98ed-7f0b57309295\" (UID: \"f45f8128-5315-4bd0-98ed-7f0b57309295\") " Nov 01 23:11:48.388891 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:48.388745 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f45f8128-5315-4bd0-98ed-7f0b57309295-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f45f8128-5315-4bd0-98ed-7f0b57309295-metrics-certs\") pod \"f45f8128-5315-4bd0-98ed-7f0b57309295\" (UID: \"f45f8128-5315-4bd0-98ed-7f0b57309295\") " Nov 01 23:11:48.389155 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:48.389125 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f45f8128-5315-4bd0-98ed-7f0b57309295-openshift-service-ca" pod "f45f8128-5315-4bd0-98ed-7f0b57309295" (UID: "f45f8128-5315-4bd0-98ed-7f0b57309295"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:48.391153 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:48.391130 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f45f8128-5315-4bd0-98ed-7f0b57309295-maas-api-tls" pod "f45f8128-5315-4bd0-98ed-7f0b57309295" (UID: "f45f8128-5315-4bd0-98ed-7f0b57309295"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:48.391480 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:48.391458 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f45f8128-5315-4bd0-98ed-7f0b57309295-kube-api-access-5jpv5" pod "f45f8128-5315-4bd0-98ed-7f0b57309295" (UID: "f45f8128-5315-4bd0-98ed-7f0b57309295"). InnerVolumeSpecName "kube-api-access-5jpv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:48.391573 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:48.391462 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f45f8128-5315-4bd0-98ed-7f0b57309295-metrics-certs" pod "f45f8128-5315-4bd0-98ed-7f0b57309295" (UID: "f45f8128-5315-4bd0-98ed-7f0b57309295"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:48.392998 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:48.392972 5011 generic.go:356] "Generic (PLEG): container finished" podID="f45f8128-5315-4bd0-98ed-7f0b57309295" containerID="689fa041d4286a4209a45cdcf7db2c59d87f58dc47d279177e7865df0c4fe695" exitCode=0 Nov 01 23:11:48.393100 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:48.393012 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb" Nov 01 23:11:48.393100 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:48.393051 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb" event={"ID":"f45f8128-5315-4bd0-98ed-7f0b57309295","Type":"ContainerDied","Data":"689fa041d4286a4209a45cdcf7db2c59d87f58dc47d279177e7865df0c4fe695"} Nov 01 23:11:48.393201 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:48.393106 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb" event={"ID":"f45f8128-5315-4bd0-98ed-7f0b57309295","Type":"ContainerDied","Data":"336e297f7a841e3f94bb07b0ad66b1a2de2be799213410d689a1a4fadc0bc7e4"} Nov 01 23:11:48.393201 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:48.393137 5011 scope.go:122] "RemoveContainer" containerID="689fa041d4286a4209a45cdcf7db2c59d87f58dc47d279177e7865df0c4fe695" Nov 01 23:11:48.415456 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:48.415436 5011 scope.go:122] "RemoveContainer" containerID="689fa041d4286a4209a45cdcf7db2c59d87f58dc47d279177e7865df0c4fe695" Nov 01 23:11:48.415762 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:11:48.415745 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"689fa041d4286a4209a45cdcf7db2c59d87f58dc47d279177e7865df0c4fe695\": container with ID starting with 689fa041d4286a4209a45cdcf7db2c59d87f58dc47d279177e7865df0c4fe695 not found: ID does not exist" containerID="689fa041d4286a4209a45cdcf7db2c59d87f58dc47d279177e7865df0c4fe695" Nov 01 23:11:48.415873 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:48.415769 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"689fa041d4286a4209a45cdcf7db2c59d87f58dc47d279177e7865df0c4fe695"} err="failed to get container status \"689fa041d4286a4209a45cdcf7db2c59d87f58dc47d279177e7865df0c4fe695\": rpc error: code = NotFound desc = could not find container \"689fa041d4286a4209a45cdcf7db2c59d87f58dc47d279177e7865df0c4fe695\": container with ID starting with 689fa041d4286a4209a45cdcf7db2c59d87f58dc47d279177e7865df0c4fe695 not found: ID does not exist" Nov 01 23:11:48.428021 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:48.427995 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb"] Nov 01 23:11:48.434022 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:48.433995 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-075771-6999d56c89-bm5kb"] Nov 01 23:11:48.490220 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:48.490185 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f45f8128-5315-4bd0-98ed-7f0b57309295-maas-api-tls\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:48.490220 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:48.490216 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f45f8128-5315-4bd0-98ed-7f0b57309295-openshift-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:48.490433 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:48.490228 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f45f8128-5315-4bd0-98ed-7f0b57309295-metrics-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:48.490433 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:48.490237 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jpv5\" (UniqueName: \"kubernetes.io/projected/f45f8128-5315-4bd0-98ed-7f0b57309295-kube-api-access-5jpv5\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:49.747321 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:49.747275 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f45f8128-5315-4bd0-98ed-7f0b57309295" path="/var/lib/kubelet/pods/f45f8128-5315-4bd0-98ed-7f0b57309295/volumes" Nov 01 23:11:50.278367 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.278324 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" containerID="cri-o://8393223ff6c758f1b996c2ab2d2277dab50ec0fee24b9ea57e54df34b7f1c8d2" gracePeriod=45 Nov 01 23:11:50.295348 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.295311 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" containerID="cri-o://9c6d68132a835dacc44926d95316b3d8c134719b0759ff7299de43db84797e9a" gracePeriod=45 Nov 01 23:11:50.412738 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.412712 5011 generic.go:356] "Generic (PLEG): container finished" podID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerID="8393223ff6c758f1b996c2ab2d2277dab50ec0fee24b9ea57e54df34b7f1c8d2" exitCode=0 Nov 01 23:11:50.412875 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.412766 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" event={"ID":"99d6cd19-de6a-43ff-8a40-5df59db6818c","Type":"ContainerDied","Data":"8393223ff6c758f1b996c2ab2d2277dab50ec0fee24b9ea57e54df34b7f1c8d2"} Nov 01 23:11:50.570273 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.570250 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 23:11:50.573463 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.573433 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 23:11:50.716358 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.716318 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-tls-certs\" (UniqueName: \"kubernetes.io/secret/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-tls-certs\") pod \"8b3ef57b-dc90-49b0-98c6-85eacd323a9b\" (UID: \"8b3ef57b-dc90-49b0-98c6-85eacd323a9b\") " Nov 01 23:11:50.716577 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.716374 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99d6cd19-de6a-43ff-8a40-5df59db6818c-tls-certs\" (UniqueName: \"kubernetes.io/secret/99d6cd19-de6a-43ff-8a40-5df59db6818c-tls-certs\") pod \"99d6cd19-de6a-43ff-8a40-5df59db6818c\" (UID: \"99d6cd19-de6a-43ff-8a40-5df59db6818c\") " Nov 01 23:11:50.716577 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.716422 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-home\" (UniqueName: \"kubernetes.io/empty-dir/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-home\") pod \"8b3ef57b-dc90-49b0-98c6-85eacd323a9b\" (UID: \"8b3ef57b-dc90-49b0-98c6-85eacd323a9b\") " Nov 01 23:11:50.716577 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.716460 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99d6cd19-de6a-43ff-8a40-5df59db6818c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99d6cd19-de6a-43ff-8a40-5df59db6818c-model-cache\") pod \"99d6cd19-de6a-43ff-8a40-5df59db6818c\" (UID: \"99d6cd19-de6a-43ff-8a40-5df59db6818c\") " Nov 01 23:11:50.716577 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.716507 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-tmp-dir\") pod \"8b3ef57b-dc90-49b0-98c6-85eacd323a9b\" (UID: \"8b3ef57b-dc90-49b0-98c6-85eacd323a9b\") " Nov 01 23:11:50.716808 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.716583 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-dshm\") pod \"8b3ef57b-dc90-49b0-98c6-85eacd323a9b\" (UID: \"8b3ef57b-dc90-49b0-98c6-85eacd323a9b\") " Nov 01 23:11:50.716808 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.716617 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-kube-api-access-7k7fs\" (UniqueName: \"kubernetes.io/projected/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-kube-api-access-7k7fs\") pod \"8b3ef57b-dc90-49b0-98c6-85eacd323a9b\" (UID: \"8b3ef57b-dc90-49b0-98c6-85eacd323a9b\") " Nov 01 23:11:50.716808 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.716649 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99d6cd19-de6a-43ff-8a40-5df59db6818c-home\" (UniqueName: \"kubernetes.io/empty-dir/99d6cd19-de6a-43ff-8a40-5df59db6818c-home\") pod \"99d6cd19-de6a-43ff-8a40-5df59db6818c\" (UID: \"99d6cd19-de6a-43ff-8a40-5df59db6818c\") " Nov 01 23:11:50.716808 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.716735 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-model-cache\") pod \"8b3ef57b-dc90-49b0-98c6-85eacd323a9b\" (UID: \"8b3ef57b-dc90-49b0-98c6-85eacd323a9b\") " Nov 01 23:11:50.716808 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.716759 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-home" pod "8b3ef57b-dc90-49b0-98c6-85eacd323a9b" (UID: "8b3ef57b-dc90-49b0-98c6-85eacd323a9b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:50.716808 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.716755 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d6cd19-de6a-43ff-8a40-5df59db6818c-model-cache" pod "99d6cd19-de6a-43ff-8a40-5df59db6818c" (UID: "99d6cd19-de6a-43ff-8a40-5df59db6818c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:50.717292 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.716768 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99d6cd19-de6a-43ff-8a40-5df59db6818c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99d6cd19-de6a-43ff-8a40-5df59db6818c-tmp-dir\") pod \"99d6cd19-de6a-43ff-8a40-5df59db6818c\" (UID: \"99d6cd19-de6a-43ff-8a40-5df59db6818c\") " Nov 01 23:11:50.717292 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.716909 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99d6cd19-de6a-43ff-8a40-5df59db6818c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/99d6cd19-de6a-43ff-8a40-5df59db6818c-dshm\") pod \"99d6cd19-de6a-43ff-8a40-5df59db6818c\" (UID: \"99d6cd19-de6a-43ff-8a40-5df59db6818c\") " Nov 01 23:11:50.717292 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.716984 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99d6cd19-de6a-43ff-8a40-5df59db6818c-kube-api-access-xx6j7\" (UniqueName: \"kubernetes.io/projected/99d6cd19-de6a-43ff-8a40-5df59db6818c-kube-api-access-xx6j7\") pod \"99d6cd19-de6a-43ff-8a40-5df59db6818c\" (UID: \"99d6cd19-de6a-43ff-8a40-5df59db6818c\") " Nov 01 23:11:50.717292 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.717021 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d6cd19-de6a-43ff-8a40-5df59db6818c-home" pod "99d6cd19-de6a-43ff-8a40-5df59db6818c" (UID: "99d6cd19-de6a-43ff-8a40-5df59db6818c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:50.717292 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.717019 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-tmp-dir" pod "8b3ef57b-dc90-49b0-98c6-85eacd323a9b" (UID: "8b3ef57b-dc90-49b0-98c6-85eacd323a9b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:50.717292 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.717038 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d6cd19-de6a-43ff-8a40-5df59db6818c-tmp-dir" pod "99d6cd19-de6a-43ff-8a40-5df59db6818c" (UID: "99d6cd19-de6a-43ff-8a40-5df59db6818c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:50.717292 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.717251 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-model-cache" pod "8b3ef57b-dc90-49b0-98c6-85eacd323a9b" (UID: "8b3ef57b-dc90-49b0-98c6-85eacd323a9b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:50.718110 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.718084 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-model-cache\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:50.718237 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.718114 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99d6cd19-de6a-43ff-8a40-5df59db6818c-tmp-dir\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:50.718237 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.718127 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-home\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:50.718237 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.718140 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99d6cd19-de6a-43ff-8a40-5df59db6818c-model-cache\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:50.718237 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.718153 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-tmp-dir\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:50.718237 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.718164 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99d6cd19-de6a-43ff-8a40-5df59db6818c-home\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:50.720193 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.720153 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-tls-certs" pod "8b3ef57b-dc90-49b0-98c6-85eacd323a9b" (UID: "8b3ef57b-dc90-49b0-98c6-85eacd323a9b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:50.723823 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.723317 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-kube-api-access-7k7fs" pod "8b3ef57b-dc90-49b0-98c6-85eacd323a9b" (UID: "8b3ef57b-dc90-49b0-98c6-85eacd323a9b"). InnerVolumeSpecName "kube-api-access-7k7fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:50.723823 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.723468 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99d6cd19-de6a-43ff-8a40-5df59db6818c-tls-certs" pod "99d6cd19-de6a-43ff-8a40-5df59db6818c" (UID: "99d6cd19-de6a-43ff-8a40-5df59db6818c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:50.724219 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.724183 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-dshm" pod "8b3ef57b-dc90-49b0-98c6-85eacd323a9b" (UID: "8b3ef57b-dc90-49b0-98c6-85eacd323a9b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:50.724921 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.724898 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d6cd19-de6a-43ff-8a40-5df59db6818c-dshm" pod "99d6cd19-de6a-43ff-8a40-5df59db6818c" (UID: "99d6cd19-de6a-43ff-8a40-5df59db6818c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:50.725368 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.725348 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d6cd19-de6a-43ff-8a40-5df59db6818c-kube-api-access-xx6j7" pod "99d6cd19-de6a-43ff-8a40-5df59db6818c" (UID: "99d6cd19-de6a-43ff-8a40-5df59db6818c"). InnerVolumeSpecName "kube-api-access-xx6j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:50.819203 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.819116 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx6j7\" (UniqueName: \"kubernetes.io/projected/99d6cd19-de6a-43ff-8a40-5df59db6818c-kube-api-access-xx6j7\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:50.819203 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.819148 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-tls-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:50.819203 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.819159 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99d6cd19-de6a-43ff-8a40-5df59db6818c-tls-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:50.819203 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.819167 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-dshm\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:50.819203 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.819177 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k7fs\" (UniqueName: \"kubernetes.io/projected/8b3ef57b-dc90-49b0-98c6-85eacd323a9b-kube-api-access-7k7fs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:50.819203 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:50.819185 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99d6cd19-de6a-43ff-8a40-5df59db6818c-dshm\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:51.430390 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:51.430351 5011 generic.go:356] "Generic (PLEG): container finished" podID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerID="9c6d68132a835dacc44926d95316b3d8c134719b0759ff7299de43db84797e9a" exitCode=0 Nov 01 23:11:51.430618 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:51.430423 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" Nov 01 23:11:51.430618 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:51.430454 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" event={"ID":"8b3ef57b-dc90-49b0-98c6-85eacd323a9b","Type":"ContainerDied","Data":"9c6d68132a835dacc44926d95316b3d8c134719b0759ff7299de43db84797e9a"} Nov 01 23:11:51.430618 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:51.430509 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw" event={"ID":"8b3ef57b-dc90-49b0-98c6-85eacd323a9b","Type":"ContainerDied","Data":"c0b08c4a87476e2085edfbb51d143b39f61e12107ee6555772457a08dcda752e"} Nov 01 23:11:51.430618 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:51.430533 5011 scope.go:122] "RemoveContainer" containerID="9c6d68132a835dacc44926d95316b3d8c134719b0759ff7299de43db84797e9a" Nov 01 23:11:51.432191 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:51.432176 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" Nov 01 23:11:51.432312 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:51.432202 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n" event={"ID":"99d6cd19-de6a-43ff-8a40-5df59db6818c","Type":"ContainerDied","Data":"c65b0d33ffbe5063c74ae890faddcb4ced268bbcbc97dcef18b9aaa874c102a2"} Nov 01 23:11:51.441457 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:51.441440 5011 scope.go:122] "RemoveContainer" containerID="9c6d68132a835dacc44926d95316b3d8c134719b0759ff7299de43db84797e9a" Nov 01 23:11:51.441852 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:11:51.441790 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c6d68132a835dacc44926d95316b3d8c134719b0759ff7299de43db84797e9a\": container with ID starting with 9c6d68132a835dacc44926d95316b3d8c134719b0759ff7299de43db84797e9a not found: ID does not exist" containerID="9c6d68132a835dacc44926d95316b3d8c134719b0759ff7299de43db84797e9a" Nov 01 23:11:51.441954 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:51.441853 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c6d68132a835dacc44926d95316b3d8c134719b0759ff7299de43db84797e9a"} err="failed to get container status \"9c6d68132a835dacc44926d95316b3d8c134719b0759ff7299de43db84797e9a\": rpc error: code = NotFound desc = could not find container \"9c6d68132a835dacc44926d95316b3d8c134719b0759ff7299de43db84797e9a\": container with ID starting with 9c6d68132a835dacc44926d95316b3d8c134719b0759ff7299de43db84797e9a not found: ID does not exist" Nov 01 23:11:51.441954 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:51.441878 5011 scope.go:122] "RemoveContainer" containerID="8393223ff6c758f1b996c2ab2d2277dab50ec0fee24b9ea57e54df34b7f1c8d2" Nov 01 23:11:51.456824 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:51.456800 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n"] Nov 01 23:11:51.460264 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:51.460246 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-f338e4/distinct-2-w1-f338e4-kserve-687bbbc8cd-9cn4n"] Nov 01 23:11:51.470344 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:51.470320 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw"] Nov 01 23:11:51.472410 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:51.472392 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-f338e4/premium-w1-f338e4-kserve-8ff4979d6-mhssw"] Nov 01 23:11:51.760200 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:51.760160 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" path="/var/lib/kubelet/pods/8b3ef57b-dc90-49b0-98c6-85eacd323a9b/volumes" Nov 01 23:11:51.760886 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:51.760863 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" path="/var/lib/kubelet/pods/99d6cd19-de6a-43ff-8a40-5df59db6818c/volumes" Nov 01 23:11:52.254586 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:52.254524 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6"] Nov 01 23:11:52.255109 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:52.254791 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6" podUID="ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb" containerName="maas-api" containerID="cri-o://37a1f61c7855eed2278050904d42015fc3c5686cae82fc42f7be097788da353b" gracePeriod=30 Nov 01 23:11:52.331525 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:52.331467 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-075771-66579df747-cbnsf" podUID="ec03babc-374f-4704-b397-4db9884f5c77" containerName="payload-processing" containerID="cri-o://0c2b7cb9bf75dadb0e28ae351067ed1ebddb0dedb1334d410d52bbd9f97934b6" gracePeriod=25 Nov 01 23:11:52.354408 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:52.354361 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-075771-66b777b8dc-cphpc" podUID="23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a" containerName="payload-pre-processing" containerID="cri-o://a5950948ce19aeb9563b993d3c68e955013658294a2dde375aacb6c1644157f4" gracePeriod=25 Nov 01 23:11:52.523062 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:52.522674 5011 generic.go:356] "Generic (PLEG): container finished" podID="ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb" containerID="37a1f61c7855eed2278050904d42015fc3c5686cae82fc42f7be097788da353b" exitCode=0 Nov 01 23:11:52.523062 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:52.522734 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6" event={"ID":"ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb","Type":"ContainerDied","Data":"37a1f61c7855eed2278050904d42015fc3c5686cae82fc42f7be097788da353b"} Nov 01 23:11:52.564059 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:52.564031 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6" Nov 01 23:11:52.651982 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:52.651951 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb-maas-api-tls\") pod \"ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb\" (UID: \"ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb\") " Nov 01 23:11:52.652160 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:52.651989 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb-kube-api-access-2tkns\" (UniqueName: \"kubernetes.io/projected/ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb-kube-api-access-2tkns\") pod \"ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb\" (UID: \"ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb\") " Nov 01 23:11:52.652160 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:52.652146 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb-openshift-service-ca\") pod \"ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb\" (UID: \"ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb\") " Nov 01 23:11:52.652236 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:52.652187 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb-metrics-certs\") pod \"ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb\" (UID: \"ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb\") " Nov 01 23:11:52.652499 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:52.652469 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb-openshift-service-ca" pod "ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb" (UID: "ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:52.652708 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:52.652685 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb-openshift-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:52.654570 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:52.654539 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb-maas-api-tls" pod "ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb" (UID: "ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:52.654692 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:52.654570 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb-kube-api-access-2tkns" pod "ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb" (UID: "ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb"). InnerVolumeSpecName "kube-api-access-2tkns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:52.654692 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:52.654593 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb-metrics-certs" pod "ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb" (UID: "ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:52.753702 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:52.753666 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb-metrics-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:52.753702 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:52.753696 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb-maas-api-tls\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:52.753702 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:52.753706 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tkns\" (UniqueName: \"kubernetes.io/projected/ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb-kube-api-access-2tkns\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:52.801942 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:52.801825 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56b833-d86bfc667-g45bg" podUID="a187f76d-d133-49a8-9614-6dcd3fe2f1b0" containerName="payload-processing" containerID="cri-o://106ffa4d58adba88e2d91f7b086247d94860c07449a4a7dc5e29508f28902ed2" gracePeriod=25 Nov 01 23:11:52.817423 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:52.817371 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56b833-556fbfdf9-snb4s" podUID="212e6260-16ba-4d02-9f5d-84cf58050476" containerName="payload-pre-processing" containerID="cri-o://acd95f134942f10f6aac371c1e1d03a5e5e865bb85c0a2cecfe2de4430bf5783" gracePeriod=25 Nov 01 23:11:53.533381 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:53.533348 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6" event={"ID":"ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb","Type":"ContainerDied","Data":"bb61d030c60135e205eee1b4c77eef5b6f5e45ab86738ea340c9305c0851cd79"} Nov 01 23:11:53.533381 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:53.533379 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6" Nov 01 23:11:53.533988 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:53.533410 5011 scope.go:122] "RemoveContainer" containerID="37a1f61c7855eed2278050904d42015fc3c5686cae82fc42f7be097788da353b" Nov 01 23:11:53.559040 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:53.559007 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6"] Nov 01 23:11:53.565853 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:53.565816 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-56b833-5847467c8f-cr5f6"] Nov 01 23:11:53.748080 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:53.748040 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb" path="/var/lib/kubelet/pods/ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb/volumes" Nov 01 23:11:55.437222 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:55.437194 5011 scope.go:122] "RemoveContainer" containerID="8e960ab1ac88f661feff2cdd7f0f3f3a9198d8bb0e7c18b95d102b1a926a498a" Nov 01 23:11:56.508154 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:56.508114 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v"] Nov 01 23:11:56.508669 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:56.508369 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v" podUID="27a3501e-233e-449f-bbfe-8df5ee0731a6" containerName="maas-api" containerID="cri-o://e90807abd205f6ce8765510b43d155924c4fb10249964f02cc0122beb6a0affc" gracePeriod=30 Nov 01 23:11:56.775612 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:56.775453 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5"] Nov 01 23:11:56.775786 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:56.775693 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" podUID="c7a2a9b4-ab92-4473-9afa-3a102b2babc4" containerName="istio-proxy" containerID="cri-o://0396a11d5b06e63d9ea5b20df3a253167c378df575f5cfb5b2dbbb574bf22d83" gracePeriod=30 Nov 01 23:11:56.785569 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:56.784872 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm"] Nov 01 23:11:56.785569 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:56.785313 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" podUID="e6f28eed-cd8b-472c-ae3a-75722591e360" containerName="istio-proxy" containerID="cri-o://ee6bc720f86ac7ba0ba89d485f08aeaab08b91d489e9f4791cada0175eeaadb5" gracePeriod=30 Nov 01 23:11:56.854357 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:56.854294 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v" Nov 01 23:11:56.923658 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:56.923623 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27a3501e-233e-449f-bbfe-8df5ee0731a6-kube-api-access-2z2bj\" (UniqueName: \"kubernetes.io/projected/27a3501e-233e-449f-bbfe-8df5ee0731a6-kube-api-access-2z2bj\") pod \"27a3501e-233e-449f-bbfe-8df5ee0731a6\" (UID: \"27a3501e-233e-449f-bbfe-8df5ee0731a6\") " Nov 01 23:11:56.923871 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:56.923689 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27a3501e-233e-449f-bbfe-8df5ee0731a6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/27a3501e-233e-449f-bbfe-8df5ee0731a6-metrics-certs\") pod \"27a3501e-233e-449f-bbfe-8df5ee0731a6\" (UID: \"27a3501e-233e-449f-bbfe-8df5ee0731a6\") " Nov 01 23:11:56.923871 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:56.923741 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27a3501e-233e-449f-bbfe-8df5ee0731a6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27a3501e-233e-449f-bbfe-8df5ee0731a6-maas-api-tls\") pod \"27a3501e-233e-449f-bbfe-8df5ee0731a6\" (UID: \"27a3501e-233e-449f-bbfe-8df5ee0731a6\") " Nov 01 23:11:56.923975 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:56.923915 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27a3501e-233e-449f-bbfe-8df5ee0731a6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27a3501e-233e-449f-bbfe-8df5ee0731a6-openshift-service-ca\") pod \"27a3501e-233e-449f-bbfe-8df5ee0731a6\" (UID: \"27a3501e-233e-449f-bbfe-8df5ee0731a6\") " Nov 01 23:11:56.924271 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:56.924243 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27a3501e-233e-449f-bbfe-8df5ee0731a6-openshift-service-ca" pod "27a3501e-233e-449f-bbfe-8df5ee0731a6" (UID: "27a3501e-233e-449f-bbfe-8df5ee0731a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:56.924408 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:56.924356 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27a3501e-233e-449f-bbfe-8df5ee0731a6-openshift-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:56.926033 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:56.926010 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27a3501e-233e-449f-bbfe-8df5ee0731a6-kube-api-access-2z2bj" pod "27a3501e-233e-449f-bbfe-8df5ee0731a6" (UID: "27a3501e-233e-449f-bbfe-8df5ee0731a6"). InnerVolumeSpecName "kube-api-access-2z2bj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:56.926638 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:56.926587 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27a3501e-233e-449f-bbfe-8df5ee0731a6-metrics-certs" pod "27a3501e-233e-449f-bbfe-8df5ee0731a6" (UID: "27a3501e-233e-449f-bbfe-8df5ee0731a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:56.926772 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:56.926667 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27a3501e-233e-449f-bbfe-8df5ee0731a6-maas-api-tls" pod "27a3501e-233e-449f-bbfe-8df5ee0731a6" (UID: "27a3501e-233e-449f-bbfe-8df5ee0731a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:57.026005 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:57.025926 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z2bj\" (UniqueName: \"kubernetes.io/projected/27a3501e-233e-449f-bbfe-8df5ee0731a6-kube-api-access-2z2bj\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:57.026005 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:57.025958 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27a3501e-233e-449f-bbfe-8df5ee0731a6-metrics-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:57.026005 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:57.025969 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27a3501e-233e-449f-bbfe-8df5ee0731a6-maas-api-tls\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:57.570137 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:57.570101 5011 generic.go:356] "Generic (PLEG): container finished" podID="27a3501e-233e-449f-bbfe-8df5ee0731a6" containerID="e90807abd205f6ce8765510b43d155924c4fb10249964f02cc0122beb6a0affc" exitCode=0 Nov 01 23:11:57.570679 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:57.570153 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v" Nov 01 23:11:57.570679 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:57.570183 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v" event={"ID":"27a3501e-233e-449f-bbfe-8df5ee0731a6","Type":"ContainerDied","Data":"e90807abd205f6ce8765510b43d155924c4fb10249964f02cc0122beb6a0affc"} Nov 01 23:11:57.570679 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:57.570235 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v" event={"ID":"27a3501e-233e-449f-bbfe-8df5ee0731a6","Type":"ContainerDied","Data":"166cf62d7b379918ad1b194972b5e0eec22173b42f86999d9d49c74e211d08f2"} Nov 01 23:11:57.570679 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:57.570277 5011 scope.go:122] "RemoveContainer" containerID="e90807abd205f6ce8765510b43d155924c4fb10249964f02cc0122beb6a0affc" Nov 01 23:11:57.581549 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:57.581523 5011 scope.go:122] "RemoveContainer" containerID="e90807abd205f6ce8765510b43d155924c4fb10249964f02cc0122beb6a0affc" Nov 01 23:11:57.581874 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:11:57.581848 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e90807abd205f6ce8765510b43d155924c4fb10249964f02cc0122beb6a0affc\": container with ID starting with e90807abd205f6ce8765510b43d155924c4fb10249964f02cc0122beb6a0affc not found: ID does not exist" containerID="e90807abd205f6ce8765510b43d155924c4fb10249964f02cc0122beb6a0affc" Nov 01 23:11:57.581940 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:57.581887 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e90807abd205f6ce8765510b43d155924c4fb10249964f02cc0122beb6a0affc"} err="failed to get container status \"e90807abd205f6ce8765510b43d155924c4fb10249964f02cc0122beb6a0affc\": rpc error: code = NotFound desc = could not find container \"e90807abd205f6ce8765510b43d155924c4fb10249964f02cc0122beb6a0affc\": container with ID starting with e90807abd205f6ce8765510b43d155924c4fb10249964f02cc0122beb6a0affc not found: ID does not exist" Nov 01 23:11:57.592775 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:57.592744 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v"] Nov 01 23:11:57.598653 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:57.598627 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-f338e4-d46694d8f-lg54v"] Nov 01 23:11:57.746430 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:57.746398 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27a3501e-233e-449f-bbfe-8df5ee0731a6" path="/var/lib/kubelet/pods/27a3501e-233e-449f-bbfe-8df5ee0731a6/volumes" Nov 01 23:11:57.875641 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:57.875549 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56b833-556fbfdf9-snb4s" podUID="212e6260-16ba-4d02-9f5d-84cf58050476" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:11:57.875812 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:57.875706 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56b833-d86bfc667-g45bg" podUID="a187f76d-d133-49a8-9614-6dcd3fe2f1b0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:11:59.099104 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:59.099070 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-075771-66b777b8dc-cphpc" podUID="23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:11:59.099619 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:59.099152 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-075771-66579df747-cbnsf" podUID="ec03babc-374f-4704-b397-4db9884f5c77" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:11:59.968919 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:59.968886 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt"] Nov 01 23:11:59.969263 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:11:59.969236 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" podUID="55ecde9f-2ac4-4791-ada0-184f9d6141f8" containerName="istio-proxy" containerID="cri-o://db051a13764c72d72b26b4f5a2333d3acc9c3ba66e31723d2d7becd52a136cfc" gracePeriod=30 Nov 01 23:12:02.068547 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.068478 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:12:02.128099 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.128064 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:12:02.190037 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.190001 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istio-token\" (UniqueName: \"kubernetes.io/projected/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istio-token\") pod \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " Nov 01 23:12:02.190254 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.190064 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6f28eed-cd8b-472c-ae3a-75722591e360-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6f28eed-cd8b-472c-ae3a-75722591e360-istiod-ca-cert\") pod \"e6f28eed-cd8b-472c-ae3a-75722591e360\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " Nov 01 23:12:02.190254 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.190103 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-credential-socket\") pod \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " Nov 01 23:12:02.190254 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.190134 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-istio-envoy\") pod \"e6f28eed-cd8b-472c-ae3a-75722591e360\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " Nov 01 23:12:02.190254 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.190208 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istio-data\") pod \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " Nov 01 23:12:02.190254 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.190249 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6f28eed-cd8b-472c-ae3a-75722591e360-kube-api-access-nq9bc\" (UniqueName: \"kubernetes.io/projected/e6f28eed-cd8b-472c-ae3a-75722591e360-kube-api-access-nq9bc\") pod \"e6f28eed-cd8b-472c-ae3a-75722591e360\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " Nov 01 23:12:02.190582 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.190282 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-workload-certs\") pod \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " Nov 01 23:12:02.190582 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.190317 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istio-podinfo\") pod \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " Nov 01 23:12:02.190582 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.190363 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-kube-api-access-952p6\" (UniqueName: \"kubernetes.io/projected/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-kube-api-access-952p6\") pod \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " Nov 01 23:12:02.190582 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.190407 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-credential-socket\") pod \"e6f28eed-cd8b-472c-ae3a-75722591e360\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " Nov 01 23:12:02.190582 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.190437 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-workload-socket\") pod \"e6f28eed-cd8b-472c-ae3a-75722591e360\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " Nov 01 23:12:02.190582 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.190465 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-istio-data\") pod \"e6f28eed-cd8b-472c-ae3a-75722591e360\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " Nov 01 23:12:02.190582 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.190503 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6f28eed-cd8b-472c-ae3a-75722591e360-istiod-ca-cert" pod "e6f28eed-cd8b-472c-ae3a-75722591e360" (UID: "e6f28eed-cd8b-472c-ae3a-75722591e360"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:02.190582 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.190525 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istio-envoy\") pod \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " Nov 01 23:12:02.190582 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.190565 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-workload-certs\") pod \"e6f28eed-cd8b-472c-ae3a-75722591e360\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " Nov 01 23:12:02.191152 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.190615 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6f28eed-cd8b-472c-ae3a-75722591e360-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6f28eed-cd8b-472c-ae3a-75722591e360-istio-podinfo\") pod \"e6f28eed-cd8b-472c-ae3a-75722591e360\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " Nov 01 23:12:02.191152 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.190668 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-workload-socket\") pod \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " Nov 01 23:12:02.191152 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.190732 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istio-data" pod "c7a2a9b4-ab92-4473-9afa-3a102b2babc4" (UID: "c7a2a9b4-ab92-4473-9afa-3a102b2babc4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:02.191152 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.190738 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6f28eed-cd8b-472c-ae3a-75722591e360-istio-token\" (UniqueName: \"kubernetes.io/projected/e6f28eed-cd8b-472c-ae3a-75722591e360-istio-token\") pod \"e6f28eed-cd8b-472c-ae3a-75722591e360\" (UID: \"e6f28eed-cd8b-472c-ae3a-75722591e360\") " Nov 01 23:12:02.191152 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.190881 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istiod-ca-cert\") pod \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\" (UID: \"c7a2a9b4-ab92-4473-9afa-3a102b2babc4\") " Nov 01 23:12:02.191152 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.190921 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-istio-data" pod "e6f28eed-cd8b-472c-ae3a-75722591e360" (UID: "e6f28eed-cd8b-472c-ae3a-75722591e360"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:02.191152 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.191121 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-credential-socket" pod "c7a2a9b4-ab92-4473-9afa-3a102b2babc4" (UID: "c7a2a9b4-ab92-4473-9afa-3a102b2babc4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:02.191537 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.191484 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-credential-socket" pod "e6f28eed-cd8b-472c-ae3a-75722591e360" (UID: "e6f28eed-cd8b-472c-ae3a-75722591e360"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:02.191723 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.191700 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-workload-certs" pod "e6f28eed-cd8b-472c-ae3a-75722591e360" (UID: "e6f28eed-cd8b-472c-ae3a-75722591e360"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:02.191826 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.191801 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istiod-ca-cert" pod "c7a2a9b4-ab92-4473-9afa-3a102b2babc4" (UID: "c7a2a9b4-ab92-4473-9afa-3a102b2babc4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:02.193092 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.192182 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-workload-socket" pod "e6f28eed-cd8b-472c-ae3a-75722591e360" (UID: "e6f28eed-cd8b-472c-ae3a-75722591e360"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:02.193092 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.192357 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.193092 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.192377 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.193092 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.192393 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6f28eed-cd8b-472c-ae3a-75722591e360-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.193092 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.192408 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.193092 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.192422 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.193092 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.192435 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.193092 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.192450 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.193092 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.192464 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.193092 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.192684 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-workload-certs" pod "c7a2a9b4-ab92-4473-9afa-3a102b2babc4" (UID: "c7a2a9b4-ab92-4473-9afa-3a102b2babc4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:02.194417 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.193639 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-workload-socket" pod "c7a2a9b4-ab92-4473-9afa-3a102b2babc4" (UID: "c7a2a9b4-ab92-4473-9afa-3a102b2babc4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:02.194920 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.194893 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6f28eed-cd8b-472c-ae3a-75722591e360-istio-podinfo" pod "e6f28eed-cd8b-472c-ae3a-75722591e360" (UID: "e6f28eed-cd8b-472c-ae3a-75722591e360"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:12:02.195054 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.194976 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6f28eed-cd8b-472c-ae3a-75722591e360-kube-api-access-nq9bc" pod "e6f28eed-cd8b-472c-ae3a-75722591e360" (UID: "e6f28eed-cd8b-472c-ae3a-75722591e360"). InnerVolumeSpecName "kube-api-access-nq9bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:02.195054 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.195002 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6f28eed-cd8b-472c-ae3a-75722591e360-istio-token" pod "e6f28eed-cd8b-472c-ae3a-75722591e360" (UID: "e6f28eed-cd8b-472c-ae3a-75722591e360"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:02.195266 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.195237 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istio-token" pod "c7a2a9b4-ab92-4473-9afa-3a102b2babc4" (UID: "c7a2a9b4-ab92-4473-9afa-3a102b2babc4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:02.195410 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.195282 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istio-podinfo" pod "c7a2a9b4-ab92-4473-9afa-3a102b2babc4" (UID: "c7a2a9b4-ab92-4473-9afa-3a102b2babc4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:12:02.195485 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.195404 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istio-envoy" pod "c7a2a9b4-ab92-4473-9afa-3a102b2babc4" (UID: "c7a2a9b4-ab92-4473-9afa-3a102b2babc4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:02.195485 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.195443 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-kube-api-access-952p6" pod "c7a2a9b4-ab92-4473-9afa-3a102b2babc4" (UID: "c7a2a9b4-ab92-4473-9afa-3a102b2babc4"). InnerVolumeSpecName "kube-api-access-952p6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:02.196548 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.196529 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-istio-envoy" pod "e6f28eed-cd8b-472c-ae3a-75722591e360" (UID: "e6f28eed-cd8b-472c-ae3a-75722591e360"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:02.200746 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.200722 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4"] Nov 01 23:12:02.293780 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.293741 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq9bc\" (UniqueName: \"kubernetes.io/projected/e6f28eed-cd8b-472c-ae3a-75722591e360-kube-api-access-nq9bc\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.293780 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.293776 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.294011 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.293791 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.294011 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.293800 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-952p6\" (UniqueName: \"kubernetes.io/projected/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-kube-api-access-952p6\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.294011 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.293811 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.294011 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.293820 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6f28eed-cd8b-472c-ae3a-75722591e360-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.294011 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.293854 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.294011 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.293862 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6f28eed-cd8b-472c-ae3a-75722591e360-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.294011 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.293870 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7a2a9b4-ab92-4473-9afa-3a102b2babc4-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.294011 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.293878 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6f28eed-cd8b-472c-ae3a-75722591e360-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.605880 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.605825 5011 generic.go:356] "Generic (PLEG): container finished" podID="c7a2a9b4-ab92-4473-9afa-3a102b2babc4" containerID="0396a11d5b06e63d9ea5b20df3a253167c378df575f5cfb5b2dbbb574bf22d83" exitCode=0 Nov 01 23:12:02.606072 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.605955 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" Nov 01 23:12:02.606072 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.605975 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" event={"ID":"c7a2a9b4-ab92-4473-9afa-3a102b2babc4","Type":"ContainerDied","Data":"0396a11d5b06e63d9ea5b20df3a253167c378df575f5cfb5b2dbbb574bf22d83"} Nov 01 23:12:02.606072 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.606048 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5" event={"ID":"c7a2a9b4-ab92-4473-9afa-3a102b2babc4","Type":"ContainerDied","Data":"457802f1d15b451bf9faf969c1845e308010575c84d32f89c715d8cb18acedfb"} Nov 01 23:12:02.606281 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.606079 5011 scope.go:122] "RemoveContainer" containerID="0396a11d5b06e63d9ea5b20df3a253167c378df575f5cfb5b2dbbb574bf22d83" Nov 01 23:12:02.608247 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.608222 5011 generic.go:356] "Generic (PLEG): container finished" podID="e6f28eed-cd8b-472c-ae3a-75722591e360" containerID="ee6bc720f86ac7ba0ba89d485f08aeaab08b91d489e9f4791cada0175eeaadb5" exitCode=0 Nov 01 23:12:02.608461 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.608252 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" event={"ID":"e6f28eed-cd8b-472c-ae3a-75722591e360","Type":"ContainerDied","Data":"ee6bc720f86ac7ba0ba89d485f08aeaab08b91d489e9f4791cada0175eeaadb5"} Nov 01 23:12:02.608461 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.608269 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" Nov 01 23:12:02.608461 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.608289 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm" event={"ID":"e6f28eed-cd8b-472c-ae3a-75722591e360","Type":"ContainerDied","Data":"1cdec6ffb1357cf27554635cb106067700b7f47fe90921bc2701d50cab2b6b3e"} Nov 01 23:12:02.614458 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.614429 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:12:02.618529 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.618492 5011 scope.go:122] "RemoveContainer" containerID="0396a11d5b06e63d9ea5b20df3a253167c378df575f5cfb5b2dbbb574bf22d83" Nov 01 23:12:02.618945 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:12:02.618918 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0396a11d5b06e63d9ea5b20df3a253167c378df575f5cfb5b2dbbb574bf22d83\": container with ID starting with 0396a11d5b06e63d9ea5b20df3a253167c378df575f5cfb5b2dbbb574bf22d83 not found: ID does not exist" containerID="0396a11d5b06e63d9ea5b20df3a253167c378df575f5cfb5b2dbbb574bf22d83" Nov 01 23:12:02.619067 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.618956 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0396a11d5b06e63d9ea5b20df3a253167c378df575f5cfb5b2dbbb574bf22d83"} err="failed to get container status \"0396a11d5b06e63d9ea5b20df3a253167c378df575f5cfb5b2dbbb574bf22d83\": rpc error: code = NotFound desc = could not find container \"0396a11d5b06e63d9ea5b20df3a253167c378df575f5cfb5b2dbbb574bf22d83\": container with ID starting with 0396a11d5b06e63d9ea5b20df3a253167c378df575f5cfb5b2dbbb574bf22d83 not found: ID does not exist" Nov 01 23:12:02.619067 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.618982 5011 scope.go:122] "RemoveContainer" containerID="ee6bc720f86ac7ba0ba89d485f08aeaab08b91d489e9f4791cada0175eeaadb5" Nov 01 23:12:02.630635 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.630614 5011 scope.go:122] "RemoveContainer" containerID="ee6bc720f86ac7ba0ba89d485f08aeaab08b91d489e9f4791cada0175eeaadb5" Nov 01 23:12:02.630973 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:12:02.630955 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee6bc720f86ac7ba0ba89d485f08aeaab08b91d489e9f4791cada0175eeaadb5\": container with ID starting with ee6bc720f86ac7ba0ba89d485f08aeaab08b91d489e9f4791cada0175eeaadb5 not found: ID does not exist" containerID="ee6bc720f86ac7ba0ba89d485f08aeaab08b91d489e9f4791cada0175eeaadb5" Nov 01 23:12:02.631022 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.630981 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee6bc720f86ac7ba0ba89d485f08aeaab08b91d489e9f4791cada0175eeaadb5"} err="failed to get container status \"ee6bc720f86ac7ba0ba89d485f08aeaab08b91d489e9f4791cada0175eeaadb5\": rpc error: code = NotFound desc = could not find container \"ee6bc720f86ac7ba0ba89d485f08aeaab08b91d489e9f4791cada0175eeaadb5\": container with ID starting with ee6bc720f86ac7ba0ba89d485f08aeaab08b91d489e9f4791cada0175eeaadb5 not found: ID does not exist" Nov 01 23:12:02.636214 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.636187 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm"] Nov 01 23:12:02.645549 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.645520 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-nprgm"] Nov 01 23:12:02.657603 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.657572 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5"] Nov 01 23:12:02.663378 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:02.663344 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-85fx5"] Nov 01 23:12:03.747232 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:03.747199 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7a2a9b4-ab92-4473-9afa-3a102b2babc4" path="/var/lib/kubelet/pods/c7a2a9b4-ab92-4473-9afa-3a102b2babc4/volumes" Nov 01 23:12:03.747648 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:03.747635 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6f28eed-cd8b-472c-ae3a-75722591e360" path="/var/lib/kubelet/pods/e6f28eed-cd8b-472c-ae3a-75722591e360/volumes" Nov 01 23:12:05.246049 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.245981 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 23:12:05.335186 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.335150 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istio-envoy\") pod \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " Nov 01 23:12:05.335360 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.335213 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istiod-ca-cert\") pod \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " Nov 01 23:12:05.335360 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.335263 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istio-data\") pod \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " Nov 01 23:12:05.335360 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.335298 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istio-token\" (UniqueName: \"kubernetes.io/projected/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istio-token\") pod \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " Nov 01 23:12:05.335548 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.335371 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55ecde9f-2ac4-4791-ada0-184f9d6141f8-kube-api-access-knv4g\" (UniqueName: \"kubernetes.io/projected/55ecde9f-2ac4-4791-ada0-184f9d6141f8-kube-api-access-knv4g\") pod \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " Nov 01 23:12:05.335548 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.335474 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-workload-socket\") pod \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " Nov 01 23:12:05.335548 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.335536 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-credential-socket\") pod \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " Nov 01 23:12:05.335696 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.335565 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istio-podinfo\") pod \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " Nov 01 23:12:05.335696 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.335609 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-workload-certs\") pod \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\" (UID: \"55ecde9f-2ac4-4791-ada0-184f9d6141f8\") " Nov 01 23:12:05.335696 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.335581 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istio-data" pod "55ecde9f-2ac4-4791-ada0-184f9d6141f8" (UID: "55ecde9f-2ac4-4791-ada0-184f9d6141f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:05.335884 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.335764 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istiod-ca-cert" pod "55ecde9f-2ac4-4791-ada0-184f9d6141f8" (UID: "55ecde9f-2ac4-4791-ada0-184f9d6141f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:05.335884 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.335799 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-credential-socket" pod "55ecde9f-2ac4-4791-ada0-184f9d6141f8" (UID: "55ecde9f-2ac4-4791-ada0-184f9d6141f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:05.336036 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.336019 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-workload-certs" pod "55ecde9f-2ac4-4791-ada0-184f9d6141f8" (UID: "55ecde9f-2ac4-4791-ada0-184f9d6141f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:05.336103 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.336032 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-workload-socket" pod "55ecde9f-2ac4-4791-ada0-184f9d6141f8" (UID: "55ecde9f-2ac4-4791-ada0-184f9d6141f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:05.336168 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.336151 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:05.336215 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.336174 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:05.336215 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.336190 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:05.336314 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.336279 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:05.336314 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.336292 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:05.338457 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.338424 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istio-envoy" pod "55ecde9f-2ac4-4791-ada0-184f9d6141f8" (UID: "55ecde9f-2ac4-4791-ada0-184f9d6141f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:05.338609 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.338509 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ecde9f-2ac4-4791-ada0-184f9d6141f8-kube-api-access-knv4g" pod "55ecde9f-2ac4-4791-ada0-184f9d6141f8" (UID: "55ecde9f-2ac4-4791-ada0-184f9d6141f8"). InnerVolumeSpecName "kube-api-access-knv4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:05.339073 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.339044 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istio-podinfo" pod "55ecde9f-2ac4-4791-ada0-184f9d6141f8" (UID: "55ecde9f-2ac4-4791-ada0-184f9d6141f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:12:05.339189 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.339155 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istio-token" pod "55ecde9f-2ac4-4791-ada0-184f9d6141f8" (UID: "55ecde9f-2ac4-4791-ada0-184f9d6141f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:05.437965 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.437926 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:05.437965 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.437958 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knv4g\" (UniqueName: \"kubernetes.io/projected/55ecde9f-2ac4-4791-ada0-184f9d6141f8-kube-api-access-knv4g\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:05.437965 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.437970 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:05.438207 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.437979 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55ecde9f-2ac4-4791-ada0-184f9d6141f8-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:05.630327 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.630229 5011 generic.go:356] "Generic (PLEG): container finished" podID="55ecde9f-2ac4-4791-ada0-184f9d6141f8" containerID="db051a13764c72d72b26b4f5a2333d3acc9c3ba66e31723d2d7becd52a136cfc" exitCode=0 Nov 01 23:12:05.630489 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.630387 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" event={"ID":"55ecde9f-2ac4-4791-ada0-184f9d6141f8","Type":"ContainerDied","Data":"db051a13764c72d72b26b4f5a2333d3acc9c3ba66e31723d2d7becd52a136cfc"} Nov 01 23:12:05.630489 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.630461 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" event={"ID":"55ecde9f-2ac4-4791-ada0-184f9d6141f8","Type":"ContainerDied","Data":"03b4ca3c700538a047e3d2290aa73817a0e46a9cb5da3c2f19e367c9f9ed5244"} Nov 01 23:12:05.630489 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.630487 5011 scope.go:122] "RemoveContainer" containerID="db051a13764c72d72b26b4f5a2333d3acc9c3ba66e31723d2d7becd52a136cfc" Nov 01 23:12:05.630631 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.630619 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt" Nov 01 23:12:05.643546 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.643524 5011 scope.go:122] "RemoveContainer" containerID="61dcae3adeec8abf26fe5998cbd979dde6ea70f158b6ac5014f02b02dbac20a0" Nov 01 23:12:05.652954 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.652817 5011 scope.go:122] "RemoveContainer" containerID="db051a13764c72d72b26b4f5a2333d3acc9c3ba66e31723d2d7becd52a136cfc" Nov 01 23:12:05.653249 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:12:05.653224 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db051a13764c72d72b26b4f5a2333d3acc9c3ba66e31723d2d7becd52a136cfc\": container with ID starting with db051a13764c72d72b26b4f5a2333d3acc9c3ba66e31723d2d7becd52a136cfc not found: ID does not exist" containerID="db051a13764c72d72b26b4f5a2333d3acc9c3ba66e31723d2d7becd52a136cfc" Nov 01 23:12:05.653367 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.653257 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db051a13764c72d72b26b4f5a2333d3acc9c3ba66e31723d2d7becd52a136cfc"} err="failed to get container status \"db051a13764c72d72b26b4f5a2333d3acc9c3ba66e31723d2d7becd52a136cfc\": rpc error: code = NotFound desc = could not find container \"db051a13764c72d72b26b4f5a2333d3acc9c3ba66e31723d2d7becd52a136cfc\": container with ID starting with db051a13764c72d72b26b4f5a2333d3acc9c3ba66e31723d2d7becd52a136cfc not found: ID does not exist" Nov 01 23:12:05.653367 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.653279 5011 scope.go:122] "RemoveContainer" containerID="61dcae3adeec8abf26fe5998cbd979dde6ea70f158b6ac5014f02b02dbac20a0" Nov 01 23:12:05.653555 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:12:05.653531 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61dcae3adeec8abf26fe5998cbd979dde6ea70f158b6ac5014f02b02dbac20a0\": container with ID starting with 61dcae3adeec8abf26fe5998cbd979dde6ea70f158b6ac5014f02b02dbac20a0 not found: ID does not exist" containerID="61dcae3adeec8abf26fe5998cbd979dde6ea70f158b6ac5014f02b02dbac20a0" Nov 01 23:12:05.653614 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.653561 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61dcae3adeec8abf26fe5998cbd979dde6ea70f158b6ac5014f02b02dbac20a0"} err="failed to get container status \"61dcae3adeec8abf26fe5998cbd979dde6ea70f158b6ac5014f02b02dbac20a0\": rpc error: code = NotFound desc = could not find container \"61dcae3adeec8abf26fe5998cbd979dde6ea70f158b6ac5014f02b02dbac20a0\": container with ID starting with 61dcae3adeec8abf26fe5998cbd979dde6ea70f158b6ac5014f02b02dbac20a0 not found: ID does not exist" Nov 01 23:12:05.658755 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.658728 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt"] Nov 01 23:12:05.663284 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.663257 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-56b833-openshift-default-f6d9cffb9-6tpdt"] Nov 01 23:12:05.745995 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:05.745965 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55ecde9f-2ac4-4791-ada0-184f9d6141f8" path="/var/lib/kubelet/pods/55ecde9f-2ac4-4791-ada0-184f9d6141f8/volumes" Nov 01 23:12:07.875600 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:07.875558 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56b833-d86bfc667-g45bg" podUID="a187f76d-d133-49a8-9614-6dcd3fe2f1b0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:12:07.875600 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:07.875594 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56b833-556fbfdf9-snb4s" podUID="212e6260-16ba-4d02-9f5d-84cf58050476" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:12:09.099113 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.099074 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-075771-66579df747-cbnsf" podUID="ec03babc-374f-4704-b397-4db9884f5c77" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:12:09.099113 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.099097 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-075771-66b777b8dc-cphpc" podUID="23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:12:09.368011 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.367928 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-639c4fc8-6srhf"] Nov 01 23:12:09.368566 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368542 5011 cpu_manager.go:414] "containerMap: removing container" podUID="55ecde9f-2ac4-4791-ada0-184f9d6141f8" containerName="istio-proxy" Nov 01 23:12:09.368566 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368564 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55ecde9f-2ac4-4791-ada0-184f9d6141f8" containerName="istio-proxy" Nov 01 23:12:09.368776 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368578 5011 cpu_manager.go:414] "containerMap: removing container" podUID="e6f28eed-cd8b-472c-ae3a-75722591e360" containerName="istio-proxy" Nov 01 23:12:09.368776 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368583 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6f28eed-cd8b-472c-ae3a-75722591e360" containerName="istio-proxy" Nov 01 23:12:09.368776 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368591 5011 cpu_manager.go:414] "containerMap: removing container" podUID="0beecaf1-fcc2-4d74-ba44-005ecf40b355" containerName="payload-processing" Nov 01 23:12:09.368776 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368599 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0beecaf1-fcc2-4d74-ba44-005ecf40b355" containerName="payload-processing" Nov 01 23:12:09.368776 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368609 5011 cpu_manager.go:414] "containerMap: removing container" podUID="55ecde9f-2ac4-4791-ada0-184f9d6141f8" containerName="istio-proxy" Nov 01 23:12:09.368776 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368616 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55ecde9f-2ac4-4791-ada0-184f9d6141f8" containerName="istio-proxy" Nov 01 23:12:09.368776 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368632 5011 cpu_manager.go:414] "containerMap: removing container" podUID="9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729" containerName="payload-processing" Nov 01 23:12:09.368776 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368639 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729" containerName="payload-processing" Nov 01 23:12:09.368776 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368649 5011 cpu_manager.go:414] "containerMap: removing container" podUID="27a3501e-233e-449f-bbfe-8df5ee0731a6" containerName="maas-api" Nov 01 23:12:09.368776 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368657 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27a3501e-233e-449f-bbfe-8df5ee0731a6" containerName="maas-api" Nov 01 23:12:09.368776 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368668 5011 cpu_manager.go:414] "containerMap: removing container" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" Nov 01 23:12:09.368776 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368674 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" Nov 01 23:12:09.368776 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368688 5011 cpu_manager.go:414] "containerMap: removing container" podUID="80ac6525-b39e-4774-b781-0b07dc22f7ab" containerName="istio-proxy" Nov 01 23:12:09.368776 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368697 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80ac6525-b39e-4774-b781-0b07dc22f7ab" containerName="istio-proxy" Nov 01 23:12:09.368776 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368714 5011 cpu_manager.go:414] "containerMap: removing container" podUID="c7a2a9b4-ab92-4473-9afa-3a102b2babc4" containerName="istio-proxy" Nov 01 23:12:09.368776 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368721 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7a2a9b4-ab92-4473-9afa-3a102b2babc4" containerName="istio-proxy" Nov 01 23:12:09.368776 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368747 5011 cpu_manager.go:414] "containerMap: removing container" podUID="ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb" containerName="maas-api" Nov 01 23:12:09.368776 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368754 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb" containerName="maas-api" Nov 01 23:12:09.368776 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368770 5011 cpu_manager.go:414] "containerMap: removing container" podUID="efff7825-e6d6-4049-a3f2-620a4aeab8a2" containerName="istio-proxy" Nov 01 23:12:09.368776 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368778 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efff7825-e6d6-4049-a3f2-620a4aeab8a2" containerName="istio-proxy" Nov 01 23:12:09.369757 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368792 5011 cpu_manager.go:414] "containerMap: removing container" podUID="6bda6fa3-e106-47fb-91d8-1d150a0249a8" containerName="payload-pre-processing" Nov 01 23:12:09.369757 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368797 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bda6fa3-e106-47fb-91d8-1d150a0249a8" containerName="payload-pre-processing" Nov 01 23:12:09.369757 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368804 5011 cpu_manager.go:414] "containerMap: removing container" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" Nov 01 23:12:09.369757 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368808 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" Nov 01 23:12:09.369757 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368814 5011 cpu_manager.go:414] "containerMap: removing container" podUID="f45f8128-5315-4bd0-98ed-7f0b57309295" containerName="maas-api" Nov 01 23:12:09.369757 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368819 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f45f8128-5315-4bd0-98ed-7f0b57309295" containerName="maas-api" Nov 01 23:12:09.369757 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368916 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b0e4bcc-a8fd-4c57-8ae8-4b2c730dc729" containerName="payload-processing" Nov 01 23:12:09.369757 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368930 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="27a3501e-233e-449f-bbfe-8df5ee0731a6" containerName="maas-api" Nov 01 23:12:09.369757 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368939 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b3ef57b-dc90-49b0-98c6-85eacd323a9b" containerName="main" Nov 01 23:12:09.369757 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368944 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="80ac6525-b39e-4774-b781-0b07dc22f7ab" containerName="istio-proxy" Nov 01 23:12:09.369757 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368951 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7a2a9b4-ab92-4473-9afa-3a102b2babc4" containerName="istio-proxy" Nov 01 23:12:09.369757 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368957 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad981a0b-7ed9-4f4f-acc4-3009d4ca38bb" containerName="maas-api" Nov 01 23:12:09.369757 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368963 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="efff7825-e6d6-4049-a3f2-620a4aeab8a2" containerName="istio-proxy" Nov 01 23:12:09.369757 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368970 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bda6fa3-e106-47fb-91d8-1d150a0249a8" containerName="payload-pre-processing" Nov 01 23:12:09.369757 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368975 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="99d6cd19-de6a-43ff-8a40-5df59db6818c" containerName="main" Nov 01 23:12:09.369757 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368980 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="f45f8128-5315-4bd0-98ed-7f0b57309295" containerName="maas-api" Nov 01 23:12:09.369757 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368987 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="55ecde9f-2ac4-4791-ada0-184f9d6141f8" containerName="istio-proxy" Nov 01 23:12:09.369757 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.368993 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6f28eed-cd8b-472c-ae3a-75722591e360" containerName="istio-proxy" Nov 01 23:12:09.369757 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.369001 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="0beecaf1-fcc2-4d74-ba44-005ecf40b355" containerName="payload-processing" Nov 01 23:12:09.369757 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.369009 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="55ecde9f-2ac4-4791-ada0-184f9d6141f8" containerName="istio-proxy" Nov 01 23:12:09.373932 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.373911 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-639c4fc8-6srhf" Nov 01 23:12:09.381193 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.380176 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:12:09.381193 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.381043 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:12:09.383193 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.383167 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-639c4fc8-6srhf"] Nov 01 23:12:09.487261 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.487189 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86aac669-ecb7-42be-9a9f-37d59ee94e5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-639c4fc8-6srhf\" (UID: \"86aac669-ecb7-42be-9a9f-37d59ee94e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-639c4fc8-6srhf" Nov 01 23:12:09.487439 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.487307 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8qpg\" (UniqueName: \"kubernetes.io/projected/86aac669-ecb7-42be-9a9f-37d59ee94e5d-kube-api-access-g8qpg\") pod \"maas-api-revoke-sub-639c4fc8-6srhf\" (UID: \"86aac669-ecb7-42be-9a9f-37d59ee94e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-639c4fc8-6srhf" Nov 01 23:12:09.487439 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.487371 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86aac669-ecb7-42be-9a9f-37d59ee94e5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-639c4fc8-6srhf\" (UID: \"86aac669-ecb7-42be-9a9f-37d59ee94e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-639c4fc8-6srhf" Nov 01 23:12:09.588783 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.588743 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8qpg\" (UniqueName: \"kubernetes.io/projected/86aac669-ecb7-42be-9a9f-37d59ee94e5d-kube-api-access-g8qpg\") pod \"maas-api-revoke-sub-639c4fc8-6srhf\" (UID: \"86aac669-ecb7-42be-9a9f-37d59ee94e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-639c4fc8-6srhf" Nov 01 23:12:09.588961 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.588850 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86aac669-ecb7-42be-9a9f-37d59ee94e5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-639c4fc8-6srhf\" (UID: \"86aac669-ecb7-42be-9a9f-37d59ee94e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-639c4fc8-6srhf" Nov 01 23:12:09.588961 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.588905 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86aac669-ecb7-42be-9a9f-37d59ee94e5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-639c4fc8-6srhf\" (UID: \"86aac669-ecb7-42be-9a9f-37d59ee94e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-639c4fc8-6srhf" Nov 01 23:12:09.589475 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.589451 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86aac669-ecb7-42be-9a9f-37d59ee94e5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-639c4fc8-6srhf\" (UID: \"86aac669-ecb7-42be-9a9f-37d59ee94e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-639c4fc8-6srhf" Nov 01 23:12:09.591555 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.591528 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86aac669-ecb7-42be-9a9f-37d59ee94e5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-639c4fc8-6srhf\" (UID: \"86aac669-ecb7-42be-9a9f-37d59ee94e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-639c4fc8-6srhf" Nov 01 23:12:09.599855 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.599802 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8qpg\" (UniqueName: \"kubernetes.io/projected/86aac669-ecb7-42be-9a9f-37d59ee94e5d-kube-api-access-g8qpg\") pod \"maas-api-revoke-sub-639c4fc8-6srhf\" (UID: \"86aac669-ecb7-42be-9a9f-37d59ee94e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-639c4fc8-6srhf" Nov 01 23:12:09.694910 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.694867 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-639c4fc8-6srhf" Nov 01 23:12:09.972658 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:09.972318 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-639c4fc8-6srhf"] Nov 01 23:12:09.978107 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:12:09.978079 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86aac669_ecb7_42be_9a9f_37d59ee94e5d.slice/crio-c7da6269862a1855602afa59a7c26f04a76af5b61152866875fa5936bdd47796 WatchSource:0}: Error finding container c7da6269862a1855602afa59a7c26f04a76af5b61152866875fa5936bdd47796: Status 404 returned error can't find the container with id c7da6269862a1855602afa59a7c26f04a76af5b61152866875fa5936bdd47796 Nov 01 23:12:10.660964 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:10.660868 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-639c4fc8-6srhf" event={"ID":"86aac669-ecb7-42be-9a9f-37d59ee94e5d","Type":"ContainerStarted","Data":"9c26287230dbf91e213ac2715a713c326620bd971135da525ee2de1afb471aa0"} Nov 01 23:12:10.660964 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:10.660935 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-639c4fc8-6srhf" event={"ID":"86aac669-ecb7-42be-9a9f-37d59ee94e5d","Type":"ContainerStarted","Data":"c7da6269862a1855602afa59a7c26f04a76af5b61152866875fa5936bdd47796"} Nov 01 23:12:10.678434 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:10.678384 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-639c4fc8-6srhf" podStartSLOduration=1.678370985 podStartE2EDuration="1.678370985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:12:10.676618124 +0000 UTC m=+2899.081208758" watchObservedRunningTime="2026-11-01 23:12:10.678370985 +0000 UTC m=+2899.082961617" Nov 01 23:12:10.684044 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:10.684015 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs"] Nov 01 23:12:11.671771 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:11.671736 5011 generic.go:356] "Generic (PLEG): container finished" podID="86aac669-ecb7-42be-9a9f-37d59ee94e5d" containerID="9c26287230dbf91e213ac2715a713c326620bd971135da525ee2de1afb471aa0" exitCode=0 Nov 01 23:12:11.672189 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:11.671815 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-639c4fc8-6srhf" event={"ID":"86aac669-ecb7-42be-9a9f-37d59ee94e5d","Type":"ContainerDied","Data":"9c26287230dbf91e213ac2715a713c326620bd971135da525ee2de1afb471aa0"} Nov 01 23:12:12.640418 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:12.640378 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-075771-66579df747-cbnsf" Nov 01 23:12:12.681291 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:12.681254 5011 generic.go:356] "Generic (PLEG): container finished" podID="23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a" containerID="a5950948ce19aeb9563b993d3c68e955013658294a2dde375aacb6c1644157f4" exitCode=0 Nov 01 23:12:12.681774 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:12.681399 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-075771-66b777b8dc-cphpc" event={"ID":"23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a","Type":"ContainerDied","Data":"a5950948ce19aeb9563b993d3c68e955013658294a2dde375aacb6c1644157f4"} Nov 01 23:12:12.683195 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:12.683170 5011 generic.go:356] "Generic (PLEG): container finished" podID="ec03babc-374f-4704-b397-4db9884f5c77" containerID="0c2b7cb9bf75dadb0e28ae351067ed1ebddb0dedb1334d410d52bbd9f97934b6" exitCode=0 Nov 01 23:12:12.683327 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:12.683236 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-075771-66579df747-cbnsf" Nov 01 23:12:12.683327 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:12.683265 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-075771-66579df747-cbnsf" event={"ID":"ec03babc-374f-4704-b397-4db9884f5c77","Type":"ContainerDied","Data":"0c2b7cb9bf75dadb0e28ae351067ed1ebddb0dedb1334d410d52bbd9f97934b6"} Nov 01 23:12:12.683327 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:12.683321 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-075771-66579df747-cbnsf" event={"ID":"ec03babc-374f-4704-b397-4db9884f5c77","Type":"ContainerDied","Data":"3a4fb97e8bba12a654ca28739399b2aeb7881946fb791e7de6ecd13f008c01ca"} Nov 01 23:12:12.683497 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:12.683349 5011 scope.go:122] "RemoveContainer" containerID="0c2b7cb9bf75dadb0e28ae351067ed1ebddb0dedb1334d410d52bbd9f97934b6" Nov 01 23:12:12.703655 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:12.703608 5011 scope.go:122] "RemoveContainer" containerID="0c2b7cb9bf75dadb0e28ae351067ed1ebddb0dedb1334d410d52bbd9f97934b6" Nov 01 23:12:12.704747 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:12:12.704712 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c2b7cb9bf75dadb0e28ae351067ed1ebddb0dedb1334d410d52bbd9f97934b6\": container with ID starting with 0c2b7cb9bf75dadb0e28ae351067ed1ebddb0dedb1334d410d52bbd9f97934b6 not found: ID does not exist" containerID="0c2b7cb9bf75dadb0e28ae351067ed1ebddb0dedb1334d410d52bbd9f97934b6" Nov 01 23:12:12.704862 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:12.704758 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c2b7cb9bf75dadb0e28ae351067ed1ebddb0dedb1334d410d52bbd9f97934b6"} err="failed to get container status \"0c2b7cb9bf75dadb0e28ae351067ed1ebddb0dedb1334d410d52bbd9f97934b6\": rpc error: code = NotFound desc = could not find container \"0c2b7cb9bf75dadb0e28ae351067ed1ebddb0dedb1334d410d52bbd9f97934b6\": container with ID starting with 0c2b7cb9bf75dadb0e28ae351067ed1ebddb0dedb1334d410d52bbd9f97934b6 not found: ID does not exist" Nov 01 23:12:12.707778 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:12.707752 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-075771-66b777b8dc-cphpc" Nov 01 23:12:12.732453 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:12.730735 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec03babc-374f-4704-b397-4db9884f5c77-kube-api-access-vqfvw\" (UniqueName: \"kubernetes.io/projected/ec03babc-374f-4704-b397-4db9884f5c77-kube-api-access-vqfvw\") pod \"ec03babc-374f-4704-b397-4db9884f5c77\" (UID: \"ec03babc-374f-4704-b397-4db9884f5c77\") " Nov 01 23:12:12.732453 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:12.730779 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec03babc-374f-4704-b397-4db9884f5c77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec03babc-374f-4704-b397-4db9884f5c77-plugins-config-volume\") pod \"ec03babc-374f-4704-b397-4db9884f5c77\" (UID: \"ec03babc-374f-4704-b397-4db9884f5c77\") " Nov 01 23:12:12.732453 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:12.732108 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec03babc-374f-4704-b397-4db9884f5c77-plugins-config-volume" pod "ec03babc-374f-4704-b397-4db9884f5c77" (UID: "ec03babc-374f-4704-b397-4db9884f5c77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:12.738531 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:12.736165 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec03babc-374f-4704-b397-4db9884f5c77-kube-api-access-vqfvw" pod "ec03babc-374f-4704-b397-4db9884f5c77" (UID: "ec03babc-374f-4704-b397-4db9884f5c77"). InnerVolumeSpecName "kube-api-access-vqfvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:12.833103 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:12.833065 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a-kube-api-access-qmhvd\" (UniqueName: \"kubernetes.io/projected/23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a-kube-api-access-qmhvd\") pod \"23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a\" (UID: \"23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a\") " Nov 01 23:12:12.833361 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:12.833238 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a-plugins-config-volume\") pod \"23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a\" (UID: \"23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a\") " Nov 01 23:12:12.833674 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:12.833645 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqfvw\" (UniqueName: \"kubernetes.io/projected/ec03babc-374f-4704-b397-4db9884f5c77-kube-api-access-vqfvw\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:12.833985 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:12.833677 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec03babc-374f-4704-b397-4db9884f5c77-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:12.833985 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:12.833788 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a-plugins-config-volume" pod "23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a" (UID: "23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:12.835474 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:12.835432 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a-kube-api-access-qmhvd" pod "23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a" (UID: "23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a"). InnerVolumeSpecName "kube-api-access-qmhvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:12.931346 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:12.931320 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-639c4fc8-6srhf" Nov 01 23:12:12.934801 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:12.934767 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmhvd\" (UniqueName: \"kubernetes.io/projected/23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a-kube-api-access-qmhvd\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:12.934920 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:12.934806 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:13.015640 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.015604 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-075771-66579df747-cbnsf"] Nov 01 23:12:13.017813 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.017786 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-075771-66579df747-cbnsf"] Nov 01 23:12:13.035292 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.035261 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86aac669-ecb7-42be-9a9f-37d59ee94e5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86aac669-ecb7-42be-9a9f-37d59ee94e5d-maas-api-xks-ca\") pod \"86aac669-ecb7-42be-9a9f-37d59ee94e5d\" (UID: \"86aac669-ecb7-42be-9a9f-37d59ee94e5d\") " Nov 01 23:12:13.035592 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.035575 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86aac669-ecb7-42be-9a9f-37d59ee94e5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86aac669-ecb7-42be-9a9f-37d59ee94e5d-maas-api-openshift-ca\") pod \"86aac669-ecb7-42be-9a9f-37d59ee94e5d\" (UID: \"86aac669-ecb7-42be-9a9f-37d59ee94e5d\") " Nov 01 23:12:13.035777 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.035764 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86aac669-ecb7-42be-9a9f-37d59ee94e5d-kube-api-access-g8qpg\" (UniqueName: \"kubernetes.io/projected/86aac669-ecb7-42be-9a9f-37d59ee94e5d-kube-api-access-g8qpg\") pod \"86aac669-ecb7-42be-9a9f-37d59ee94e5d\" (UID: \"86aac669-ecb7-42be-9a9f-37d59ee94e5d\") " Nov 01 23:12:13.036070 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.036041 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86aac669-ecb7-42be-9a9f-37d59ee94e5d-maas-api-openshift-ca" pod "86aac669-ecb7-42be-9a9f-37d59ee94e5d" (UID: "86aac669-ecb7-42be-9a9f-37d59ee94e5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:13.036384 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.036367 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86aac669-ecb7-42be-9a9f-37d59ee94e5d-maas-api-openshift-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:13.038867 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.038822 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86aac669-ecb7-42be-9a9f-37d59ee94e5d-maas-api-xks-ca" pod "86aac669-ecb7-42be-9a9f-37d59ee94e5d" (UID: "86aac669-ecb7-42be-9a9f-37d59ee94e5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:12:13.039584 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.039553 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86aac669-ecb7-42be-9a9f-37d59ee94e5d-kube-api-access-g8qpg" pod "86aac669-ecb7-42be-9a9f-37d59ee94e5d" (UID: "86aac669-ecb7-42be-9a9f-37d59ee94e5d"). InnerVolumeSpecName "kube-api-access-g8qpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:13.073480 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.073441 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56b833-d86bfc667-g45bg" Nov 01 23:12:13.114985 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.114959 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56b833-556fbfdf9-snb4s" Nov 01 23:12:13.138154 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.138124 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8qpg\" (UniqueName: \"kubernetes.io/projected/86aac669-ecb7-42be-9a9f-37d59ee94e5d-kube-api-access-g8qpg\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:13.138309 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.138156 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86aac669-ecb7-42be-9a9f-37d59ee94e5d-maas-api-xks-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:13.239202 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.239108 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/212e6260-16ba-4d02-9f5d-84cf58050476-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/212e6260-16ba-4d02-9f5d-84cf58050476-plugins-config-volume\") pod \"212e6260-16ba-4d02-9f5d-84cf58050476\" (UID: \"212e6260-16ba-4d02-9f5d-84cf58050476\") " Nov 01 23:12:13.239202 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.239160 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/212e6260-16ba-4d02-9f5d-84cf58050476-kube-api-access-lfph6\" (UniqueName: \"kubernetes.io/projected/212e6260-16ba-4d02-9f5d-84cf58050476-kube-api-access-lfph6\") pod \"212e6260-16ba-4d02-9f5d-84cf58050476\" (UID: \"212e6260-16ba-4d02-9f5d-84cf58050476\") " Nov 01 23:12:13.239202 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.239195 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a187f76d-d133-49a8-9614-6dcd3fe2f1b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a187f76d-d133-49a8-9614-6dcd3fe2f1b0-plugins-config-volume\") pod \"a187f76d-d133-49a8-9614-6dcd3fe2f1b0\" (UID: \"a187f76d-d133-49a8-9614-6dcd3fe2f1b0\") " Nov 01 23:12:13.239515 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.239291 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a187f76d-d133-49a8-9614-6dcd3fe2f1b0-kube-api-access-wpb4p\" (UniqueName: \"kubernetes.io/projected/a187f76d-d133-49a8-9614-6dcd3fe2f1b0-kube-api-access-wpb4p\") pod \"a187f76d-d133-49a8-9614-6dcd3fe2f1b0\" (UID: \"a187f76d-d133-49a8-9614-6dcd3fe2f1b0\") " Nov 01 23:12:13.239574 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.239552 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/212e6260-16ba-4d02-9f5d-84cf58050476-plugins-config-volume" pod "212e6260-16ba-4d02-9f5d-84cf58050476" (UID: "212e6260-16ba-4d02-9f5d-84cf58050476"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:13.239628 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.239580 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a187f76d-d133-49a8-9614-6dcd3fe2f1b0-plugins-config-volume" pod "a187f76d-d133-49a8-9614-6dcd3fe2f1b0" (UID: "a187f76d-d133-49a8-9614-6dcd3fe2f1b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:13.239703 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.239682 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/212e6260-16ba-4d02-9f5d-84cf58050476-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:13.239756 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.239710 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a187f76d-d133-49a8-9614-6dcd3fe2f1b0-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:13.241511 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.241488 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/212e6260-16ba-4d02-9f5d-84cf58050476-kube-api-access-lfph6" pod "212e6260-16ba-4d02-9f5d-84cf58050476" (UID: "212e6260-16ba-4d02-9f5d-84cf58050476"). InnerVolumeSpecName "kube-api-access-lfph6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:13.241660 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.241640 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a187f76d-d133-49a8-9614-6dcd3fe2f1b0-kube-api-access-wpb4p" pod "a187f76d-d133-49a8-9614-6dcd3fe2f1b0" (UID: "a187f76d-d133-49a8-9614-6dcd3fe2f1b0"). InnerVolumeSpecName "kube-api-access-wpb4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:13.340561 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.340527 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpb4p\" (UniqueName: \"kubernetes.io/projected/a187f76d-d133-49a8-9614-6dcd3fe2f1b0-kube-api-access-wpb4p\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:13.340561 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.340556 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfph6\" (UniqueName: \"kubernetes.io/projected/212e6260-16ba-4d02-9f5d-84cf58050476-kube-api-access-lfph6\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:13.690012 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.689974 5011 generic.go:356] "Generic (PLEG): container finished" podID="a187f76d-d133-49a8-9614-6dcd3fe2f1b0" containerID="106ffa4d58adba88e2d91f7b086247d94860c07449a4a7dc5e29508f28902ed2" exitCode=0 Nov 01 23:12:13.690012 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.690006 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56b833-d86bfc667-g45bg" Nov 01 23:12:13.690523 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.690067 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56b833-d86bfc667-g45bg" event={"ID":"a187f76d-d133-49a8-9614-6dcd3fe2f1b0","Type":"ContainerDied","Data":"106ffa4d58adba88e2d91f7b086247d94860c07449a4a7dc5e29508f28902ed2"} Nov 01 23:12:13.690523 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.690114 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-56b833-d86bfc667-g45bg" event={"ID":"a187f76d-d133-49a8-9614-6dcd3fe2f1b0","Type":"ContainerDied","Data":"da49b047282cfe68b7b8436079ea5014098bcac5f7897dbebcec28762797c62c"} Nov 01 23:12:13.690523 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.690136 5011 scope.go:122] "RemoveContainer" containerID="106ffa4d58adba88e2d91f7b086247d94860c07449a4a7dc5e29508f28902ed2" Nov 01 23:12:13.691949 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.691928 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-075771-66b777b8dc-cphpc" event={"ID":"23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a","Type":"ContainerDied","Data":"1520326e83f9a8a9127ce82790063ef64e6b05da3682d8254ae022e9f83f1fec"} Nov 01 23:12:13.692067 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.691921 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-075771-66b777b8dc-cphpc" Nov 01 23:12:13.693682 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.693574 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-639c4fc8-6srhf" event={"ID":"86aac669-ecb7-42be-9a9f-37d59ee94e5d","Type":"ContainerDied","Data":"c7da6269862a1855602afa59a7c26f04a76af5b61152866875fa5936bdd47796"} Nov 01 23:12:13.693682 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.693619 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7da6269862a1855602afa59a7c26f04a76af5b61152866875fa5936bdd47796" Nov 01 23:12:13.693682 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.693642 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-639c4fc8-6srhf" Nov 01 23:12:13.696429 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.696407 5011 generic.go:356] "Generic (PLEG): container finished" podID="212e6260-16ba-4d02-9f5d-84cf58050476" containerID="acd95f134942f10f6aac371c1e1d03a5e5e865bb85c0a2cecfe2de4430bf5783" exitCode=0 Nov 01 23:12:13.696518 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.696437 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56b833-556fbfdf9-snb4s" event={"ID":"212e6260-16ba-4d02-9f5d-84cf58050476","Type":"ContainerDied","Data":"acd95f134942f10f6aac371c1e1d03a5e5e865bb85c0a2cecfe2de4430bf5783"} Nov 01 23:12:13.696518 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.696471 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56b833-556fbfdf9-snb4s" event={"ID":"212e6260-16ba-4d02-9f5d-84cf58050476","Type":"ContainerDied","Data":"79f935dd3ee2969fbc5d779836056e537e12ba630184dad6754b3ae00f97f5f6"} Nov 01 23:12:13.696585 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.696522 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56b833-556fbfdf9-snb4s" Nov 01 23:12:13.704070 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.703817 5011 scope.go:122] "RemoveContainer" containerID="106ffa4d58adba88e2d91f7b086247d94860c07449a4a7dc5e29508f28902ed2" Nov 01 23:12:13.704182 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:12:13.704160 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"106ffa4d58adba88e2d91f7b086247d94860c07449a4a7dc5e29508f28902ed2\": container with ID starting with 106ffa4d58adba88e2d91f7b086247d94860c07449a4a7dc5e29508f28902ed2 not found: ID does not exist" containerID="106ffa4d58adba88e2d91f7b086247d94860c07449a4a7dc5e29508f28902ed2" Nov 01 23:12:13.704227 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.704194 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"106ffa4d58adba88e2d91f7b086247d94860c07449a4a7dc5e29508f28902ed2"} err="failed to get container status \"106ffa4d58adba88e2d91f7b086247d94860c07449a4a7dc5e29508f28902ed2\": rpc error: code = NotFound desc = could not find container \"106ffa4d58adba88e2d91f7b086247d94860c07449a4a7dc5e29508f28902ed2\": container with ID starting with 106ffa4d58adba88e2d91f7b086247d94860c07449a4a7dc5e29508f28902ed2 not found: ID does not exist" Nov 01 23:12:13.704227 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.704219 5011 scope.go:122] "RemoveContainer" containerID="a5950948ce19aeb9563b993d3c68e955013658294a2dde375aacb6c1644157f4" Nov 01 23:12:13.720362 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.713959 5011 scope.go:122] "RemoveContainer" containerID="acd95f134942f10f6aac371c1e1d03a5e5e865bb85c0a2cecfe2de4430bf5783" Nov 01 23:12:13.743215 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.738607 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-075771-66b777b8dc-cphpc"] Nov 01 23:12:13.748886 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.748857 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec03babc-374f-4704-b397-4db9884f5c77" path="/var/lib/kubelet/pods/ec03babc-374f-4704-b397-4db9884f5c77/volumes" Nov 01 23:12:13.749153 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.749140 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-075771-66b777b8dc-cphpc"] Nov 01 23:12:13.749621 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.749608 5011 scope.go:122] "RemoveContainer" containerID="acd95f134942f10f6aac371c1e1d03a5e5e865bb85c0a2cecfe2de4430bf5783" Nov 01 23:12:13.749969 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:12:13.749952 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acd95f134942f10f6aac371c1e1d03a5e5e865bb85c0a2cecfe2de4430bf5783\": container with ID starting with acd95f134942f10f6aac371c1e1d03a5e5e865bb85c0a2cecfe2de4430bf5783 not found: ID does not exist" containerID="acd95f134942f10f6aac371c1e1d03a5e5e865bb85c0a2cecfe2de4430bf5783" Nov 01 23:12:13.750029 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.749976 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acd95f134942f10f6aac371c1e1d03a5e5e865bb85c0a2cecfe2de4430bf5783"} err="failed to get container status \"acd95f134942f10f6aac371c1e1d03a5e5e865bb85c0a2cecfe2de4430bf5783\": rpc error: code = NotFound desc = could not find container \"acd95f134942f10f6aac371c1e1d03a5e5e865bb85c0a2cecfe2de4430bf5783\": container with ID starting with acd95f134942f10f6aac371c1e1d03a5e5e865bb85c0a2cecfe2de4430bf5783 not found: ID does not exist" Nov 01 23:12:13.756904 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.756879 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-56b833-d86bfc667-g45bg"] Nov 01 23:12:13.762284 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.762266 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-56b833-d86bfc667-g45bg"] Nov 01 23:12:13.774211 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.774191 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56b833-556fbfdf9-snb4s"] Nov 01 23:12:13.776953 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:13.776932 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-56b833-556fbfdf9-snb4s"] Nov 01 23:12:15.685123 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:15.685085 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs" podUID="af426678-7013-404d-a85c-555c7abba8c7" containerName="payload-pre-processing" containerID="cri-o://c3611b76aac7926b7ff46797106c1d74fcfac16be77ac022e876f5ff6c4bc880" gracePeriod=25 Nov 01 23:12:15.711449 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:15.711409 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs" podUID="af426678-7013-404d-a85c-555c7abba8c7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:12:15.745801 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:15.745768 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="212e6260-16ba-4d02-9f5d-84cf58050476" path="/var/lib/kubelet/pods/212e6260-16ba-4d02-9f5d-84cf58050476/volumes" Nov 01 23:12:15.746231 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:15.746216 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a" path="/var/lib/kubelet/pods/23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a/volumes" Nov 01 23:12:15.746555 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:15.746543 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a187f76d-d133-49a8-9614-6dcd3fe2f1b0" path="/var/lib/kubelet/pods/a187f76d-d133-49a8-9614-6dcd3fe2f1b0/volumes" Nov 01 23:12:16.960282 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:12:16.960243 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:12:17.223505 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:17.223409 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" containerID="cri-o://83bbb81a37f8939a60fe1bd91c6130afd7bf90a8645009482ce904108b9f8f38" gracePeriod=45 Nov 01 23:12:17.484916 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:17.484856 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 23:12:17.584698 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:17.584670 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-tls-certs\") pod \"0a41f431-3a44-4fff-9b38-ad4e52c4fe33\" (UID: \"0a41f431-3a44-4fff-9b38-ad4e52c4fe33\") " Nov 01 23:12:17.584885 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:17.584737 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-model-cache\") pod \"0a41f431-3a44-4fff-9b38-ad4e52c4fe33\" (UID: \"0a41f431-3a44-4fff-9b38-ad4e52c4fe33\") " Nov 01 23:12:17.584885 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:17.584762 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-tmp-dir\") pod \"0a41f431-3a44-4fff-9b38-ad4e52c4fe33\" (UID: \"0a41f431-3a44-4fff-9b38-ad4e52c4fe33\") " Nov 01 23:12:17.584885 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:17.584785 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-home\" (UniqueName: \"kubernetes.io/empty-dir/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-home\") pod \"0a41f431-3a44-4fff-9b38-ad4e52c4fe33\" (UID: \"0a41f431-3a44-4fff-9b38-ad4e52c4fe33\") " Nov 01 23:12:17.585058 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:17.584889 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-kube-api-access-ztvkx\" (UniqueName: \"kubernetes.io/projected/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-kube-api-access-ztvkx\") pod \"0a41f431-3a44-4fff-9b38-ad4e52c4fe33\" (UID: \"0a41f431-3a44-4fff-9b38-ad4e52c4fe33\") " Nov 01 23:12:17.585058 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:17.584984 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-dshm\") pod \"0a41f431-3a44-4fff-9b38-ad4e52c4fe33\" (UID: \"0a41f431-3a44-4fff-9b38-ad4e52c4fe33\") " Nov 01 23:12:17.585058 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:17.585035 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-model-cache" pod "0a41f431-3a44-4fff-9b38-ad4e52c4fe33" (UID: "0a41f431-3a44-4fff-9b38-ad4e52c4fe33"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:17.585220 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:17.585059 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-tmp-dir" pod "0a41f431-3a44-4fff-9b38-ad4e52c4fe33" (UID: "0a41f431-3a44-4fff-9b38-ad4e52c4fe33"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:17.585220 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:17.585152 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-home" pod "0a41f431-3a44-4fff-9b38-ad4e52c4fe33" (UID: "0a41f431-3a44-4fff-9b38-ad4e52c4fe33"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:17.585379 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:17.585360 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-tmp-dir\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:17.585416 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:17.585382 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-home\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:17.585416 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:17.585392 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-model-cache\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:17.587074 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:17.587048 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-tls-certs" pod "0a41f431-3a44-4fff-9b38-ad4e52c4fe33" (UID: "0a41f431-3a44-4fff-9b38-ad4e52c4fe33"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:12:17.587319 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:17.587293 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-dshm" pod "0a41f431-3a44-4fff-9b38-ad4e52c4fe33" (UID: "0a41f431-3a44-4fff-9b38-ad4e52c4fe33"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:17.587433 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:17.587324 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-kube-api-access-ztvkx" pod "0a41f431-3a44-4fff-9b38-ad4e52c4fe33" (UID: "0a41f431-3a44-4fff-9b38-ad4e52c4fe33"). InnerVolumeSpecName "kube-api-access-ztvkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:17.686942 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:17.686910 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztvkx\" (UniqueName: \"kubernetes.io/projected/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-kube-api-access-ztvkx\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:17.686942 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:17.686940 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-dshm\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:17.686942 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:17.686950 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a41f431-3a44-4fff-9b38-ad4e52c4fe33-tls-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:17.727289 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:17.727250 5011 generic.go:356] "Generic (PLEG): container finished" podID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerID="83bbb81a37f8939a60fe1bd91c6130afd7bf90a8645009482ce904108b9f8f38" exitCode=0 Nov 01 23:12:17.727492 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:17.727293 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" Nov 01 23:12:17.727492 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:17.727304 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" event={"ID":"0a41f431-3a44-4fff-9b38-ad4e52c4fe33","Type":"ContainerDied","Data":"83bbb81a37f8939a60fe1bd91c6130afd7bf90a8645009482ce904108b9f8f38"} Nov 01 23:12:17.727492 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:17.727343 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4" event={"ID":"0a41f431-3a44-4fff-9b38-ad4e52c4fe33","Type":"ContainerDied","Data":"462e24422b9a5fe7181f7619fddcbdf0532ab93eeceffea2fd75d15243c723d5"} Nov 01 23:12:17.727492 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:17.727363 5011 scope.go:122] "RemoveContainer" containerID="83bbb81a37f8939a60fe1bd91c6130afd7bf90a8645009482ce904108b9f8f38" Nov 01 23:12:17.740666 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:17.740637 5011 scope.go:122] "RemoveContainer" containerID="83bbb81a37f8939a60fe1bd91c6130afd7bf90a8645009482ce904108b9f8f38" Nov 01 23:12:17.741044 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:12:17.741011 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83bbb81a37f8939a60fe1bd91c6130afd7bf90a8645009482ce904108b9f8f38\": container with ID starting with 83bbb81a37f8939a60fe1bd91c6130afd7bf90a8645009482ce904108b9f8f38 not found: ID does not exist" containerID="83bbb81a37f8939a60fe1bd91c6130afd7bf90a8645009482ce904108b9f8f38" Nov 01 23:12:17.741153 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:17.741056 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83bbb81a37f8939a60fe1bd91c6130afd7bf90a8645009482ce904108b9f8f38"} err="failed to get container status \"83bbb81a37f8939a60fe1bd91c6130afd7bf90a8645009482ce904108b9f8f38\": rpc error: code = NotFound desc = could not find container \"83bbb81a37f8939a60fe1bd91c6130afd7bf90a8645009482ce904108b9f8f38\": container with ID starting with 83bbb81a37f8939a60fe1bd91c6130afd7bf90a8645009482ce904108b9f8f38 not found: ID does not exist" Nov 01 23:12:17.756666 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:17.756573 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4"] Nov 01 23:12:17.758581 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:17.758559 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-e26733/distinct-w5-e26733-kserve-6c8548bbdb-lddj4"] Nov 01 23:12:17.934857 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:17.928006 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb"] Nov 01 23:12:17.934857 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:17.928275 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb" podUID="ae1a941e-5ef1-46b4-8948-70b23ff5b300" containerName="maas-api" containerID="cri-o://7f3b3a97f0e009d95a013d186fc5db1bed6ebe0e8a6d9ce4a887403010ef8a57" gracePeriod=30 Nov 01 23:12:18.422918 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:18.418692 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb" Nov 01 23:12:18.598643 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:18.598550 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae1a941e-5ef1-46b4-8948-70b23ff5b300-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae1a941e-5ef1-46b4-8948-70b23ff5b300-openshift-service-ca\") pod \"ae1a941e-5ef1-46b4-8948-70b23ff5b300\" (UID: \"ae1a941e-5ef1-46b4-8948-70b23ff5b300\") " Nov 01 23:12:18.598643 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:18.598590 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae1a941e-5ef1-46b4-8948-70b23ff5b300-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae1a941e-5ef1-46b4-8948-70b23ff5b300-maas-api-tls\") pod \"ae1a941e-5ef1-46b4-8948-70b23ff5b300\" (UID: \"ae1a941e-5ef1-46b4-8948-70b23ff5b300\") " Nov 01 23:12:18.598643 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:18.598625 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae1a941e-5ef1-46b4-8948-70b23ff5b300-kube-api-access-w55rf\" (UniqueName: \"kubernetes.io/projected/ae1a941e-5ef1-46b4-8948-70b23ff5b300-kube-api-access-w55rf\") pod \"ae1a941e-5ef1-46b4-8948-70b23ff5b300\" (UID: \"ae1a941e-5ef1-46b4-8948-70b23ff5b300\") " Nov 01 23:12:18.599006 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:18.598713 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae1a941e-5ef1-46b4-8948-70b23ff5b300-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae1a941e-5ef1-46b4-8948-70b23ff5b300-metrics-certs\") pod \"ae1a941e-5ef1-46b4-8948-70b23ff5b300\" (UID: \"ae1a941e-5ef1-46b4-8948-70b23ff5b300\") " Nov 01 23:12:18.599080 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:18.599015 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae1a941e-5ef1-46b4-8948-70b23ff5b300-openshift-service-ca" pod "ae1a941e-5ef1-46b4-8948-70b23ff5b300" (UID: "ae1a941e-5ef1-46b4-8948-70b23ff5b300"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:18.601076 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:18.601039 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae1a941e-5ef1-46b4-8948-70b23ff5b300-kube-api-access-w55rf" pod "ae1a941e-5ef1-46b4-8948-70b23ff5b300" (UID: "ae1a941e-5ef1-46b4-8948-70b23ff5b300"). InnerVolumeSpecName "kube-api-access-w55rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:18.601206 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:18.601083 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae1a941e-5ef1-46b4-8948-70b23ff5b300-metrics-certs" pod "ae1a941e-5ef1-46b4-8948-70b23ff5b300" (UID: "ae1a941e-5ef1-46b4-8948-70b23ff5b300"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:12:18.601206 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:18.601136 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae1a941e-5ef1-46b4-8948-70b23ff5b300-maas-api-tls" pod "ae1a941e-5ef1-46b4-8948-70b23ff5b300" (UID: "ae1a941e-5ef1-46b4-8948-70b23ff5b300"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:12:18.700527 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:18.700472 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w55rf\" (UniqueName: \"kubernetes.io/projected/ae1a941e-5ef1-46b4-8948-70b23ff5b300-kube-api-access-w55rf\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:18.700527 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:18.700528 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae1a941e-5ef1-46b4-8948-70b23ff5b300-metrics-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:18.700527 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:18.700545 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae1a941e-5ef1-46b4-8948-70b23ff5b300-openshift-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:18.700883 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:18.700560 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae1a941e-5ef1-46b4-8948-70b23ff5b300-maas-api-tls\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:18.735141 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:18.735100 5011 generic.go:356] "Generic (PLEG): container finished" podID="ae1a941e-5ef1-46b4-8948-70b23ff5b300" containerID="7f3b3a97f0e009d95a013d186fc5db1bed6ebe0e8a6d9ce4a887403010ef8a57" exitCode=0 Nov 01 23:12:18.735326 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:18.735144 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb" Nov 01 23:12:18.735326 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:18.735184 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb" event={"ID":"ae1a941e-5ef1-46b4-8948-70b23ff5b300","Type":"ContainerDied","Data":"7f3b3a97f0e009d95a013d186fc5db1bed6ebe0e8a6d9ce4a887403010ef8a57"} Nov 01 23:12:18.735326 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:18.735267 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb" event={"ID":"ae1a941e-5ef1-46b4-8948-70b23ff5b300","Type":"ContainerDied","Data":"3301762b154df0c216dd60d60e01ec952fa4c93d5b42b45d8e4b73c30d340a7f"} Nov 01 23:12:18.735326 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:18.735291 5011 scope.go:122] "RemoveContainer" containerID="7f3b3a97f0e009d95a013d186fc5db1bed6ebe0e8a6d9ce4a887403010ef8a57" Nov 01 23:12:18.746985 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:18.746961 5011 scope.go:122] "RemoveContainer" containerID="7f3b3a97f0e009d95a013d186fc5db1bed6ebe0e8a6d9ce4a887403010ef8a57" Nov 01 23:12:18.747501 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:12:18.747478 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f3b3a97f0e009d95a013d186fc5db1bed6ebe0e8a6d9ce4a887403010ef8a57\": container with ID starting with 7f3b3a97f0e009d95a013d186fc5db1bed6ebe0e8a6d9ce4a887403010ef8a57 not found: ID does not exist" containerID="7f3b3a97f0e009d95a013d186fc5db1bed6ebe0e8a6d9ce4a887403010ef8a57" Nov 01 23:12:18.747622 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:18.747514 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f3b3a97f0e009d95a013d186fc5db1bed6ebe0e8a6d9ce4a887403010ef8a57"} err="failed to get container status \"7f3b3a97f0e009d95a013d186fc5db1bed6ebe0e8a6d9ce4a887403010ef8a57\": rpc error: code = NotFound desc = could not find container \"7f3b3a97f0e009d95a013d186fc5db1bed6ebe0e8a6d9ce4a887403010ef8a57\": container with ID starting with 7f3b3a97f0e009d95a013d186fc5db1bed6ebe0e8a6d9ce4a887403010ef8a57 not found: ID does not exist" Nov 01 23:12:18.765160 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:18.765127 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb"] Nov 01 23:12:18.771444 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:18.771417 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dc635e-7dbc45bd57-8cghb"] Nov 01 23:12:19.745323 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:19.745292 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" path="/var/lib/kubelet/pods/0a41f431-3a44-4fff-9b38-ad4e52c4fe33/volumes" Nov 01 23:12:19.745758 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:19.745743 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae1a941e-5ef1-46b4-8948-70b23ff5b300" path="/var/lib/kubelet/pods/ae1a941e-5ef1-46b4-8948-70b23ff5b300/volumes" Nov 01 23:12:25.710767 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:25.710726 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs" podUID="af426678-7013-404d-a85c-555c7abba8c7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:12:26.698803 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:26.698757 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd"] Nov 01 23:12:26.699093 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:26.699043 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" podUID="9643e76a-dfa8-42b1-8270-cdd219ceceac" containerName="istio-proxy" containerID="cri-o://e8fcd90bc8481db73938c7b09dbce57de9b21a0f2cd849e8dbe31bb96482e0d9" gracePeriod=30 Nov 01 23:12:30.194252 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:30.194214 5011 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:12:30.194746 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:30.194271 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" podUID="9643e76a-dfa8-42b1-8270-cdd219ceceac" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:12:31.896867 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:31.896805 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:12:31.939131 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:31.939094 5011 generic.go:356] "Generic (PLEG): container finished" podID="9643e76a-dfa8-42b1-8270-cdd219ceceac" containerID="e8fcd90bc8481db73938c7b09dbce57de9b21a0f2cd849e8dbe31bb96482e0d9" exitCode=0 Nov 01 23:12:31.939359 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:31.939182 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" event={"ID":"9643e76a-dfa8-42b1-8270-cdd219ceceac","Type":"ContainerDied","Data":"e8fcd90bc8481db73938c7b09dbce57de9b21a0f2cd849e8dbe31bb96482e0d9"} Nov 01 23:12:32.082333 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.082298 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t"] Nov 01 23:12:32.093583 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.093442 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:12:32.103574 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.103551 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr"] Nov 01 23:12:32.151080 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.151045 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-workload-certs\") pod \"9643e76a-dfa8-42b1-8270-cdd219ceceac\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " Nov 01 23:12:32.151285 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.151104 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-istio-envoy\") pod \"9643e76a-dfa8-42b1-8270-cdd219ceceac\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " Nov 01 23:12:32.151285 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.151157 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9643e76a-dfa8-42b1-8270-cdd219ceceac-istio-token\" (UniqueName: \"kubernetes.io/projected/9643e76a-dfa8-42b1-8270-cdd219ceceac-istio-token\") pod \"9643e76a-dfa8-42b1-8270-cdd219ceceac\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " Nov 01 23:12:32.151285 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.151195 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-workload-socket\") pod \"9643e76a-dfa8-42b1-8270-cdd219ceceac\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " Nov 01 23:12:32.151285 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.151232 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-credential-socket\") pod \"9643e76a-dfa8-42b1-8270-cdd219ceceac\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " Nov 01 23:12:32.151285 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.151271 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9643e76a-dfa8-42b1-8270-cdd219ceceac-kube-api-access-xdxdw\" (UniqueName: \"kubernetes.io/projected/9643e76a-dfa8-42b1-8270-cdd219ceceac-kube-api-access-xdxdw\") pod \"9643e76a-dfa8-42b1-8270-cdd219ceceac\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " Nov 01 23:12:32.151555 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.151309 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-istio-data\") pod \"9643e76a-dfa8-42b1-8270-cdd219ceceac\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " Nov 01 23:12:32.151555 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.151373 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9643e76a-dfa8-42b1-8270-cdd219ceceac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9643e76a-dfa8-42b1-8270-cdd219ceceac-istio-podinfo\") pod \"9643e76a-dfa8-42b1-8270-cdd219ceceac\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " Nov 01 23:12:32.151555 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.151482 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9643e76a-dfa8-42b1-8270-cdd219ceceac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9643e76a-dfa8-42b1-8270-cdd219ceceac-istiod-ca-cert\") pod \"9643e76a-dfa8-42b1-8270-cdd219ceceac\" (UID: \"9643e76a-dfa8-42b1-8270-cdd219ceceac\") " Nov 01 23:12:32.152320 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.152289 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9643e76a-dfa8-42b1-8270-cdd219ceceac-istiod-ca-cert" pod "9643e76a-dfa8-42b1-8270-cdd219ceceac" (UID: "9643e76a-dfa8-42b1-8270-cdd219ceceac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:32.152612 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.152589 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-workload-certs" pod "9643e76a-dfa8-42b1-8270-cdd219ceceac" (UID: "9643e76a-dfa8-42b1-8270-cdd219ceceac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:32.153270 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.153218 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-credential-socket" pod "9643e76a-dfa8-42b1-8270-cdd219ceceac" (UID: "9643e76a-dfa8-42b1-8270-cdd219ceceac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:32.154214 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.154043 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-workload-socket" pod "9643e76a-dfa8-42b1-8270-cdd219ceceac" (UID: "9643e76a-dfa8-42b1-8270-cdd219ceceac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:32.154844 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.154383 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-istio-data" pod "9643e76a-dfa8-42b1-8270-cdd219ceceac" (UID: "9643e76a-dfa8-42b1-8270-cdd219ceceac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:32.155914 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.155450 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-istio-envoy" pod "9643e76a-dfa8-42b1-8270-cdd219ceceac" (UID: "9643e76a-dfa8-42b1-8270-cdd219ceceac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:32.159353 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.159328 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9643e76a-dfa8-42b1-8270-cdd219ceceac-istio-token" pod "9643e76a-dfa8-42b1-8270-cdd219ceceac" (UID: "9643e76a-dfa8-42b1-8270-cdd219ceceac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:32.161475 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.161438 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9643e76a-dfa8-42b1-8270-cdd219ceceac-kube-api-access-xdxdw" pod "9643e76a-dfa8-42b1-8270-cdd219ceceac" (UID: "9643e76a-dfa8-42b1-8270-cdd219ceceac"). InnerVolumeSpecName "kube-api-access-xdxdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:32.163958 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.163917 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9643e76a-dfa8-42b1-8270-cdd219ceceac-istio-podinfo" pod "9643e76a-dfa8-42b1-8270-cdd219ceceac" (UID: "9643e76a-dfa8-42b1-8270-cdd219ceceac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:12:32.252816 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.252783 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:32.252816 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.252812 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9643e76a-dfa8-42b1-8270-cdd219ceceac-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:32.252816 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.252822 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:32.252816 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.252845 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:32.253123 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.252855 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdxdw\" (UniqueName: \"kubernetes.io/projected/9643e76a-dfa8-42b1-8270-cdd219ceceac-kube-api-access-xdxdw\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:32.253123 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.252865 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:32.253123 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.252873 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9643e76a-dfa8-42b1-8270-cdd219ceceac-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:32.253123 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.252882 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9643e76a-dfa8-42b1-8270-cdd219ceceac-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:32.253123 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.252890 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9643e76a-dfa8-42b1-8270-cdd219ceceac-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:32.946399 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.946363 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" Nov 01 23:12:32.946399 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.946376 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd" event={"ID":"9643e76a-dfa8-42b1-8270-cdd219ceceac","Type":"ContainerDied","Data":"1a85fb7a48ef01726a33785e2d22cbc856658d951077caf58245d36571dcdb3c"} Nov 01 23:12:32.946969 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.946443 5011 scope.go:122] "RemoveContainer" containerID="e8fcd90bc8481db73938c7b09dbce57de9b21a0f2cd849e8dbe31bb96482e0d9" Nov 01 23:12:32.971383 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.971354 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd"] Nov 01 23:12:32.976410 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:32.976385 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-rqbgd"] Nov 01 23:12:33.746054 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:33.746021 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9643e76a-dfa8-42b1-8270-cdd219ceceac" path="/var/lib/kubelet/pods/9643e76a-dfa8-42b1-8270-cdd219ceceac/volumes" Nov 01 23:12:34.556100 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:34.556066 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7"] Nov 01 23:12:34.556628 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:34.556310 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" podUID="b0d043d9-49a8-42fb-872c-cd5479590b1f" containerName="istio-proxy" containerID="cri-o://9d66921f3d5de5fe46e049a80484bf43c6a5aa3c1f8802caa53dd027827571db" gracePeriod=30 Nov 01 23:12:35.710812 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:12:35.710783 5011 logging.go:55] [core] [Channel #10301 SubChannel #10302]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.53:50052", ServerName: "10.133.0.53:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.53:50052: connect: connection refused" Nov 01 23:12:35.711314 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:35.711280 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz"] Nov 01 23:12:35.711681 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:35.711648 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" podUID="db213002-5fde-4c7a-ba7e-6762c011b4a9" containerName="istio-proxy" containerID="cri-o://3f59012b57498e6be5618e057870b6a1110f62f48568e383f9a69ab8aecb38f1" gracePeriod=30 Nov 01 23:12:35.971228 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:35.971196 5011 generic.go:356] "Generic (PLEG): container finished" podID="af426678-7013-404d-a85c-555c7abba8c7" containerID="c3611b76aac7926b7ff46797106c1d74fcfac16be77ac022e876f5ff6c4bc880" exitCode=0 Nov 01 23:12:35.971380 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:35.971285 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs" event={"ID":"af426678-7013-404d-a85c-555c7abba8c7","Type":"ContainerDied","Data":"c3611b76aac7926b7ff46797106c1d74fcfac16be77ac022e876f5ff6c4bc880"} Nov 01 23:12:35.971380 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:35.971356 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs" event={"ID":"af426678-7013-404d-a85c-555c7abba8c7","Type":"ContainerDied","Data":"208fdc1d286b6fcd172ca3d9087be4919cb90eb69801504114c09923862ff6af"} Nov 01 23:12:35.971504 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:35.971380 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="208fdc1d286b6fcd172ca3d9087be4919cb90eb69801504114c09923862ff6af" Nov 01 23:12:35.987370 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:35.987343 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs" Nov 01 23:12:36.098004 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:36.097959 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af426678-7013-404d-a85c-555c7abba8c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af426678-7013-404d-a85c-555c7abba8c7-plugins-config-volume\") pod \"af426678-7013-404d-a85c-555c7abba8c7\" (UID: \"af426678-7013-404d-a85c-555c7abba8c7\") " Nov 01 23:12:36.098202 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:36.098016 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af426678-7013-404d-a85c-555c7abba8c7-kube-api-access-8rqxx\" (UniqueName: \"kubernetes.io/projected/af426678-7013-404d-a85c-555c7abba8c7-kube-api-access-8rqxx\") pod \"af426678-7013-404d-a85c-555c7abba8c7\" (UID: \"af426678-7013-404d-a85c-555c7abba8c7\") " Nov 01 23:12:36.098440 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:36.098411 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af426678-7013-404d-a85c-555c7abba8c7-plugins-config-volume" pod "af426678-7013-404d-a85c-555c7abba8c7" (UID: "af426678-7013-404d-a85c-555c7abba8c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:36.098554 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:36.098525 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af426678-7013-404d-a85c-555c7abba8c7-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:36.101094 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:36.101064 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af426678-7013-404d-a85c-555c7abba8c7-kube-api-access-8rqxx" pod "af426678-7013-404d-a85c-555c7abba8c7" (UID: "af426678-7013-404d-a85c-555c7abba8c7"). InnerVolumeSpecName "kube-api-access-8rqxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:36.199810 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:36.199772 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rqxx\" (UniqueName: \"kubernetes.io/projected/af426678-7013-404d-a85c-555c7abba8c7-kube-api-access-8rqxx\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:36.710338 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:36.710297 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs" podUID="af426678-7013-404d-a85c-555c7abba8c7" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.53:50052\" within 1s: context deadline exceeded" Nov 01 23:12:36.710521 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:36.710381 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs" Nov 01 23:12:36.976817 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:36.976739 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs" Nov 01 23:12:36.997425 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:36.997389 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs"] Nov 01 23:12:37.003990 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.003960 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-6ac182-6fcc77fd67-cjgxs"] Nov 01 23:12:37.398447 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.398357 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e5f47e-6b9bd944c5-jbtnp"] Nov 01 23:12:37.571960 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.571925 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4517e7a2-d2vvv"] Nov 01 23:12:37.572416 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.572399 5011 cpu_manager.go:414] "containerMap: removing container" podUID="af426678-7013-404d-a85c-555c7abba8c7" containerName="payload-pre-processing" Nov 01 23:12:37.572502 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.572418 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af426678-7013-404d-a85c-555c7abba8c7" containerName="payload-pre-processing" Nov 01 23:12:37.572502 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.572429 5011 cpu_manager.go:414] "containerMap: removing container" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" Nov 01 23:12:37.572502 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.572434 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" Nov 01 23:12:37.572502 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.572443 5011 cpu_manager.go:414] "containerMap: removing container" podUID="a187f76d-d133-49a8-9614-6dcd3fe2f1b0" containerName="payload-processing" Nov 01 23:12:37.572502 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.572451 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a187f76d-d133-49a8-9614-6dcd3fe2f1b0" containerName="payload-processing" Nov 01 23:12:37.572502 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.572462 5011 cpu_manager.go:414] "containerMap: removing container" podUID="9643e76a-dfa8-42b1-8270-cdd219ceceac" containerName="istio-proxy" Nov 01 23:12:37.572502 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.572469 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9643e76a-dfa8-42b1-8270-cdd219ceceac" containerName="istio-proxy" Nov 01 23:12:37.572502 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.572482 5011 cpu_manager.go:414] "containerMap: removing container" podUID="212e6260-16ba-4d02-9f5d-84cf58050476" containerName="payload-pre-processing" Nov 01 23:12:37.572502 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.572490 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="212e6260-16ba-4d02-9f5d-84cf58050476" containerName="payload-pre-processing" Nov 01 23:12:37.572802 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.572516 5011 cpu_manager.go:414] "containerMap: removing container" podUID="86aac669-ecb7-42be-9a9f-37d59ee94e5d" containerName="revoke-keys" Nov 01 23:12:37.572802 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.572523 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86aac669-ecb7-42be-9a9f-37d59ee94e5d" containerName="revoke-keys" Nov 01 23:12:37.572802 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.572554 5011 cpu_manager.go:414] "containerMap: removing container" podUID="ec03babc-374f-4704-b397-4db9884f5c77" containerName="payload-processing" Nov 01 23:12:37.572802 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.572559 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec03babc-374f-4704-b397-4db9884f5c77" containerName="payload-processing" Nov 01 23:12:37.572802 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.572565 5011 cpu_manager.go:414] "containerMap: removing container" podUID="23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a" containerName="payload-pre-processing" Nov 01 23:12:37.572802 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.572571 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a" containerName="payload-pre-processing" Nov 01 23:12:37.572802 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.572578 5011 cpu_manager.go:414] "containerMap: removing container" podUID="ae1a941e-5ef1-46b4-8948-70b23ff5b300" containerName="maas-api" Nov 01 23:12:37.572802 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.572582 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae1a941e-5ef1-46b4-8948-70b23ff5b300" containerName="maas-api" Nov 01 23:12:37.572802 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.572649 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="86aac669-ecb7-42be-9a9f-37d59ee94e5d" containerName="revoke-keys" Nov 01 23:12:37.572802 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.572661 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec03babc-374f-4704-b397-4db9884f5c77" containerName="payload-processing" Nov 01 23:12:37.572802 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.572666 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d1d2d2-11d8-4ca4-b08a-7ba3713b3b1a" containerName="payload-pre-processing" Nov 01 23:12:37.572802 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.572673 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae1a941e-5ef1-46b4-8948-70b23ff5b300" containerName="maas-api" Nov 01 23:12:37.572802 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.572679 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="af426678-7013-404d-a85c-555c7abba8c7" containerName="payload-pre-processing" Nov 01 23:12:37.572802 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.572683 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a41f431-3a44-4fff-9b38-ad4e52c4fe33" containerName="main" Nov 01 23:12:37.572802 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.572688 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="a187f76d-d133-49a8-9614-6dcd3fe2f1b0" containerName="payload-processing" Nov 01 23:12:37.572802 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.572694 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="9643e76a-dfa8-42b1-8270-cdd219ceceac" containerName="istio-proxy" Nov 01 23:12:37.572802 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.572699 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="212e6260-16ba-4d02-9f5d-84cf58050476" containerName="payload-pre-processing" Nov 01 23:12:37.575826 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.575809 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4517e7a2-d2vvv" Nov 01 23:12:37.587373 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.587341 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4517e7a2-d2vvv"] Nov 01 23:12:37.670330 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.670300 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:12:37.670546 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.670342 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:12:37.714733 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.714694 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh5s9\" (UniqueName: \"kubernetes.io/projected/80493f88-4b1b-4d2c-acb5-f956a719af86-kube-api-access-gh5s9\") pod \"maas-api-revoke-sub-4517e7a2-d2vvv\" (UID: \"80493f88-4b1b-4d2c-acb5-f956a719af86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4517e7a2-d2vvv" Nov 01 23:12:37.714733 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.714738 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80493f88-4b1b-4d2c-acb5-f956a719af86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4517e7a2-d2vvv\" (UID: \"80493f88-4b1b-4d2c-acb5-f956a719af86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4517e7a2-d2vvv" Nov 01 23:12:37.715005 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.714880 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80493f88-4b1b-4d2c-acb5-f956a719af86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4517e7a2-d2vvv\" (UID: \"80493f88-4b1b-4d2c-acb5-f956a719af86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4517e7a2-d2vvv" Nov 01 23:12:37.745087 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.745057 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af426678-7013-404d-a85c-555c7abba8c7" path="/var/lib/kubelet/pods/af426678-7013-404d-a85c-555c7abba8c7/volumes" Nov 01 23:12:37.816115 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.816080 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80493f88-4b1b-4d2c-acb5-f956a719af86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4517e7a2-d2vvv\" (UID: \"80493f88-4b1b-4d2c-acb5-f956a719af86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4517e7a2-d2vvv" Nov 01 23:12:37.816320 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.816188 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh5s9\" (UniqueName: \"kubernetes.io/projected/80493f88-4b1b-4d2c-acb5-f956a719af86-kube-api-access-gh5s9\") pod \"maas-api-revoke-sub-4517e7a2-d2vvv\" (UID: \"80493f88-4b1b-4d2c-acb5-f956a719af86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4517e7a2-d2vvv" Nov 01 23:12:37.816320 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.816226 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80493f88-4b1b-4d2c-acb5-f956a719af86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4517e7a2-d2vvv\" (UID: \"80493f88-4b1b-4d2c-acb5-f956a719af86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4517e7a2-d2vvv" Nov 01 23:12:37.816800 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.816778 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80493f88-4b1b-4d2c-acb5-f956a719af86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4517e7a2-d2vvv\" (UID: \"80493f88-4b1b-4d2c-acb5-f956a719af86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4517e7a2-d2vvv" Nov 01 23:12:37.818608 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.818589 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80493f88-4b1b-4d2c-acb5-f956a719af86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4517e7a2-d2vvv\" (UID: \"80493f88-4b1b-4d2c-acb5-f956a719af86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4517e7a2-d2vvv" Nov 01 23:12:37.826895 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.826874 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh5s9\" (UniqueName: \"kubernetes.io/projected/80493f88-4b1b-4d2c-acb5-f956a719af86-kube-api-access-gh5s9\") pod \"maas-api-revoke-sub-4517e7a2-d2vvv\" (UID: \"80493f88-4b1b-4d2c-acb5-f956a719af86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4517e7a2-d2vvv" Nov 01 23:12:37.839672 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.839644 5011 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:12:37.839784 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.839695 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" podUID="b0d043d9-49a8-42fb-872c-cd5479590b1f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:12:37.891400 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:37.891370 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4517e7a2-d2vvv" Nov 01 23:12:38.172744 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:38.169795 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4517e7a2-d2vvv"] Nov 01 23:12:38.988864 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:38.988814 5011 generic.go:356] "Generic (PLEG): container finished" podID="80493f88-4b1b-4d2c-acb5-f956a719af86" containerID="daee16e3ed3a1cde2226dcc632208a3de02f6904826c7b8267d38727d16c2d84" exitCode=0 Nov 01 23:12:38.988998 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:38.988872 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4517e7a2-d2vvv" event={"ID":"80493f88-4b1b-4d2c-acb5-f956a719af86","Type":"ContainerDied","Data":"daee16e3ed3a1cde2226dcc632208a3de02f6904826c7b8267d38727d16c2d84"} Nov 01 23:12:38.988998 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:38.988926 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4517e7a2-d2vvv" event={"ID":"80493f88-4b1b-4d2c-acb5-f956a719af86","Type":"ContainerStarted","Data":"f152077af25a660813552ffd0b1ec3e8eea657b6b78870058c1abf6b8ff74ba0"} Nov 01 23:12:39.824245 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:39.824220 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 23:12:39.938083 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:39.938051 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0d043d9-49a8-42fb-872c-cd5479590b1f-istio-token\" (UniqueName: \"kubernetes.io/projected/b0d043d9-49a8-42fb-872c-cd5479590b1f-istio-token\") pod \"b0d043d9-49a8-42fb-872c-cd5479590b1f\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " Nov 01 23:12:39.938294 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:39.938129 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-istio-data\") pod \"b0d043d9-49a8-42fb-872c-cd5479590b1f\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " Nov 01 23:12:39.938294 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:39.938238 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-credential-socket\") pod \"b0d043d9-49a8-42fb-872c-cd5479590b1f\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " Nov 01 23:12:39.938445 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:39.938358 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0d043d9-49a8-42fb-872c-cd5479590b1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0d043d9-49a8-42fb-872c-cd5479590b1f-istio-podinfo\") pod \"b0d043d9-49a8-42fb-872c-cd5479590b1f\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " Nov 01 23:12:39.938445 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:39.938384 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-istio-envoy\") pod \"b0d043d9-49a8-42fb-872c-cd5479590b1f\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " Nov 01 23:12:39.938445 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:39.938409 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-workload-certs\") pod \"b0d043d9-49a8-42fb-872c-cd5479590b1f\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " Nov 01 23:12:39.938591 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:39.938440 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-istio-data" pod "b0d043d9-49a8-42fb-872c-cd5479590b1f" (UID: "b0d043d9-49a8-42fb-872c-cd5479590b1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:39.938591 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:39.938443 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0d043d9-49a8-42fb-872c-cd5479590b1f-kube-api-access-9smxt\" (UniqueName: \"kubernetes.io/projected/b0d043d9-49a8-42fb-872c-cd5479590b1f-kube-api-access-9smxt\") pod \"b0d043d9-49a8-42fb-872c-cd5479590b1f\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " Nov 01 23:12:39.938591 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:39.938466 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-credential-socket" pod "b0d043d9-49a8-42fb-872c-cd5479590b1f" (UID: "b0d043d9-49a8-42fb-872c-cd5479590b1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:39.938591 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:39.938531 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-workload-socket\") pod \"b0d043d9-49a8-42fb-872c-cd5479590b1f\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " Nov 01 23:12:39.938591 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:39.938572 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0d043d9-49a8-42fb-872c-cd5479590b1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0d043d9-49a8-42fb-872c-cd5479590b1f-istiod-ca-cert\") pod \"b0d043d9-49a8-42fb-872c-cd5479590b1f\" (UID: \"b0d043d9-49a8-42fb-872c-cd5479590b1f\") " Nov 01 23:12:39.938816 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:39.938701 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-workload-certs" pod "b0d043d9-49a8-42fb-872c-cd5479590b1f" (UID: "b0d043d9-49a8-42fb-872c-cd5479590b1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:39.938908 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:39.938865 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-workload-socket" pod "b0d043d9-49a8-42fb-872c-cd5479590b1f" (UID: "b0d043d9-49a8-42fb-872c-cd5479590b1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:39.939017 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:39.938998 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0d043d9-49a8-42fb-872c-cd5479590b1f-istiod-ca-cert" pod "b0d043d9-49a8-42fb-872c-cd5479590b1f" (UID: "b0d043d9-49a8-42fb-872c-cd5479590b1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:39.939096 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:39.939081 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:39.939322 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:39.939100 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:39.939322 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:39.939114 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:39.939322 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:39.939127 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:39.939322 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:39.939140 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0d043d9-49a8-42fb-872c-cd5479590b1f-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:39.940882 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:39.940856 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0d043d9-49a8-42fb-872c-cd5479590b1f-istio-podinfo" pod "b0d043d9-49a8-42fb-872c-cd5479590b1f" (UID: "b0d043d9-49a8-42fb-872c-cd5479590b1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:12:39.941191 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:39.941167 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-istio-envoy" pod "b0d043d9-49a8-42fb-872c-cd5479590b1f" (UID: "b0d043d9-49a8-42fb-872c-cd5479590b1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:39.941285 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:39.941197 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d043d9-49a8-42fb-872c-cd5479590b1f-istio-token" pod "b0d043d9-49a8-42fb-872c-cd5479590b1f" (UID: "b0d043d9-49a8-42fb-872c-cd5479590b1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:39.941285 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:39.941220 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d043d9-49a8-42fb-872c-cd5479590b1f-kube-api-access-9smxt" pod "b0d043d9-49a8-42fb-872c-cd5479590b1f" (UID: "b0d043d9-49a8-42fb-872c-cd5479590b1f"). InnerVolumeSpecName "kube-api-access-9smxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:39.996347 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:39.996298 5011 generic.go:356] "Generic (PLEG): container finished" podID="b0d043d9-49a8-42fb-872c-cd5479590b1f" containerID="9d66921f3d5de5fe46e049a80484bf43c6a5aa3c1f8802caa53dd027827571db" exitCode=0 Nov 01 23:12:39.996528 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:39.996356 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" Nov 01 23:12:39.996528 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:39.996383 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" event={"ID":"b0d043d9-49a8-42fb-872c-cd5479590b1f","Type":"ContainerDied","Data":"9d66921f3d5de5fe46e049a80484bf43c6a5aa3c1f8802caa53dd027827571db"} Nov 01 23:12:39.996528 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:39.996448 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7" event={"ID":"b0d043d9-49a8-42fb-872c-cd5479590b1f","Type":"ContainerDied","Data":"413ba639462ecdb6ec20a55deec3a17e84c39ccadb1d20d1fe50c537e01fa053"} Nov 01 23:12:39.996528 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:39.996483 5011 scope.go:122] "RemoveContainer" containerID="9d66921f3d5de5fe46e049a80484bf43c6a5aa3c1f8802caa53dd027827571db" Nov 01 23:12:40.009194 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:40.009166 5011 scope.go:122] "RemoveContainer" containerID="3d39ef4a83cc4951b1891e0c772456f2b9022de0303a5895c61b9e1bb550f43d" Nov 01 23:12:40.020895 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:40.020870 5011 scope.go:122] "RemoveContainer" containerID="9d66921f3d5de5fe46e049a80484bf43c6a5aa3c1f8802caa53dd027827571db" Nov 01 23:12:40.021324 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:12:40.021292 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d66921f3d5de5fe46e049a80484bf43c6a5aa3c1f8802caa53dd027827571db\": container with ID starting with 9d66921f3d5de5fe46e049a80484bf43c6a5aa3c1f8802caa53dd027827571db not found: ID does not exist" containerID="9d66921f3d5de5fe46e049a80484bf43c6a5aa3c1f8802caa53dd027827571db" Nov 01 23:12:40.021639 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:40.021337 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d66921f3d5de5fe46e049a80484bf43c6a5aa3c1f8802caa53dd027827571db"} err="failed to get container status \"9d66921f3d5de5fe46e049a80484bf43c6a5aa3c1f8802caa53dd027827571db\": rpc error: code = NotFound desc = could not find container \"9d66921f3d5de5fe46e049a80484bf43c6a5aa3c1f8802caa53dd027827571db\": container with ID starting with 9d66921f3d5de5fe46e049a80484bf43c6a5aa3c1f8802caa53dd027827571db not found: ID does not exist" Nov 01 23:12:40.021639 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:40.021364 5011 scope.go:122] "RemoveContainer" containerID="3d39ef4a83cc4951b1891e0c772456f2b9022de0303a5895c61b9e1bb550f43d" Nov 01 23:12:40.021781 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:40.021643 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7"] Nov 01 23:12:40.021962 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:12:40.021926 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d39ef4a83cc4951b1891e0c772456f2b9022de0303a5895c61b9e1bb550f43d\": container with ID starting with 3d39ef4a83cc4951b1891e0c772456f2b9022de0303a5895c61b9e1bb550f43d not found: ID does not exist" containerID="3d39ef4a83cc4951b1891e0c772456f2b9022de0303a5895c61b9e1bb550f43d" Nov 01 23:12:40.022071 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:40.021966 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d39ef4a83cc4951b1891e0c772456f2b9022de0303a5895c61b9e1bb550f43d"} err="failed to get container status \"3d39ef4a83cc4951b1891e0c772456f2b9022de0303a5895c61b9e1bb550f43d\": rpc error: code = NotFound desc = could not find container \"3d39ef4a83cc4951b1891e0c772456f2b9022de0303a5895c61b9e1bb550f43d\": container with ID starting with 3d39ef4a83cc4951b1891e0c772456f2b9022de0303a5895c61b9e1bb550f43d not found: ID does not exist" Nov 01 23:12:40.024007 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:40.023977 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-5k7s7"] Nov 01 23:12:40.040732 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:40.040703 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0d043d9-49a8-42fb-872c-cd5479590b1f-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:40.040732 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:40.040729 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0d043d9-49a8-42fb-872c-cd5479590b1f-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:40.040962 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:40.040743 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9smxt\" (UniqueName: \"kubernetes.io/projected/b0d043d9-49a8-42fb-872c-cd5479590b1f-kube-api-access-9smxt\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:40.040962 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:40.040756 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0d043d9-49a8-42fb-872c-cd5479590b1f-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:40.142380 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:40.142346 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4517e7a2-d2vvv" Nov 01 23:12:40.243699 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:40.243621 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80493f88-4b1b-4d2c-acb5-f956a719af86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80493f88-4b1b-4d2c-acb5-f956a719af86-maas-api-xks-ca\") pod \"80493f88-4b1b-4d2c-acb5-f956a719af86\" (UID: \"80493f88-4b1b-4d2c-acb5-f956a719af86\") " Nov 01 23:12:40.243699 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:40.243662 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80493f88-4b1b-4d2c-acb5-f956a719af86-kube-api-access-gh5s9\" (UniqueName: \"kubernetes.io/projected/80493f88-4b1b-4d2c-acb5-f956a719af86-kube-api-access-gh5s9\") pod \"80493f88-4b1b-4d2c-acb5-f956a719af86\" (UID: \"80493f88-4b1b-4d2c-acb5-f956a719af86\") " Nov 01 23:12:40.243935 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:40.243720 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80493f88-4b1b-4d2c-acb5-f956a719af86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80493f88-4b1b-4d2c-acb5-f956a719af86-maas-api-openshift-ca\") pod \"80493f88-4b1b-4d2c-acb5-f956a719af86\" (UID: \"80493f88-4b1b-4d2c-acb5-f956a719af86\") " Nov 01 23:12:40.244180 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:40.244156 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80493f88-4b1b-4d2c-acb5-f956a719af86-maas-api-openshift-ca" pod "80493f88-4b1b-4d2c-acb5-f956a719af86" (UID: "80493f88-4b1b-4d2c-acb5-f956a719af86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:40.245927 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:40.245901 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80493f88-4b1b-4d2c-acb5-f956a719af86-maas-api-xks-ca" pod "80493f88-4b1b-4d2c-acb5-f956a719af86" (UID: "80493f88-4b1b-4d2c-acb5-f956a719af86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:12:40.246051 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:40.246028 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80493f88-4b1b-4d2c-acb5-f956a719af86-kube-api-access-gh5s9" pod "80493f88-4b1b-4d2c-acb5-f956a719af86" (UID: "80493f88-4b1b-4d2c-acb5-f956a719af86"). InnerVolumeSpecName "kube-api-access-gh5s9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:40.345406 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:40.345366 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80493f88-4b1b-4d2c-acb5-f956a719af86-maas-api-xks-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:40.345406 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:40.345404 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh5s9\" (UniqueName: \"kubernetes.io/projected/80493f88-4b1b-4d2c-acb5-f956a719af86-kube-api-access-gh5s9\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:40.345406 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:40.345415 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80493f88-4b1b-4d2c-acb5-f956a719af86-maas-api-openshift-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:40.987861 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:40.987756 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 23:12:41.016426 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.016380 5011 generic.go:356] "Generic (PLEG): container finished" podID="db213002-5fde-4c7a-ba7e-6762c011b4a9" containerID="3f59012b57498e6be5618e057870b6a1110f62f48568e383f9a69ab8aecb38f1" exitCode=0 Nov 01 23:12:41.016615 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.016482 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" event={"ID":"db213002-5fde-4c7a-ba7e-6762c011b4a9","Type":"ContainerDied","Data":"3f59012b57498e6be5618e057870b6a1110f62f48568e383f9a69ab8aecb38f1"} Nov 01 23:12:41.016615 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.016535 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" event={"ID":"db213002-5fde-4c7a-ba7e-6762c011b4a9","Type":"ContainerDied","Data":"06251651c92740284eb773426239cad97af4168319d051cf5d1cdbc00769cfd9"} Nov 01 23:12:41.016615 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.016573 5011 scope.go:122] "RemoveContainer" containerID="3f59012b57498e6be5618e057870b6a1110f62f48568e383f9a69ab8aecb38f1" Nov 01 23:12:41.016807 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.016679 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz" Nov 01 23:12:41.020039 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.019399 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4517e7a2-d2vvv" event={"ID":"80493f88-4b1b-4d2c-acb5-f956a719af86","Type":"ContainerDied","Data":"f152077af25a660813552ffd0b1ec3e8eea657b6b78870058c1abf6b8ff74ba0"} Nov 01 23:12:41.020039 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.019453 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f152077af25a660813552ffd0b1ec3e8eea657b6b78870058c1abf6b8ff74ba0" Nov 01 23:12:41.020039 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.019525 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4517e7a2-d2vvv" Nov 01 23:12:41.030682 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.030572 5011 scope.go:122] "RemoveContainer" containerID="3f59012b57498e6be5618e057870b6a1110f62f48568e383f9a69ab8aecb38f1" Nov 01 23:12:41.031105 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:12:41.031075 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f59012b57498e6be5618e057870b6a1110f62f48568e383f9a69ab8aecb38f1\": container with ID starting with 3f59012b57498e6be5618e057870b6a1110f62f48568e383f9a69ab8aecb38f1 not found: ID does not exist" containerID="3f59012b57498e6be5618e057870b6a1110f62f48568e383f9a69ab8aecb38f1" Nov 01 23:12:41.031226 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.031116 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f59012b57498e6be5618e057870b6a1110f62f48568e383f9a69ab8aecb38f1"} err="failed to get container status \"3f59012b57498e6be5618e057870b6a1110f62f48568e383f9a69ab8aecb38f1\": rpc error: code = NotFound desc = could not find container \"3f59012b57498e6be5618e057870b6a1110f62f48568e383f9a69ab8aecb38f1\": container with ID starting with 3f59012b57498e6be5618e057870b6a1110f62f48568e383f9a69ab8aecb38f1 not found: ID does not exist" Nov 01 23:12:41.154543 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.154498 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-istio-data\") pod \"db213002-5fde-4c7a-ba7e-6762c011b4a9\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " Nov 01 23:12:41.154757 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.154570 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db213002-5fde-4c7a-ba7e-6762c011b4a9-istio-token\" (UniqueName: \"kubernetes.io/projected/db213002-5fde-4c7a-ba7e-6762c011b4a9-istio-token\") pod \"db213002-5fde-4c7a-ba7e-6762c011b4a9\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " Nov 01 23:12:41.154757 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.154601 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-workload-socket\") pod \"db213002-5fde-4c7a-ba7e-6762c011b4a9\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " Nov 01 23:12:41.154757 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.154630 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db213002-5fde-4c7a-ba7e-6762c011b4a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db213002-5fde-4c7a-ba7e-6762c011b4a9-istiod-ca-cert\") pod \"db213002-5fde-4c7a-ba7e-6762c011b4a9\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " Nov 01 23:12:41.154757 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.154656 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-workload-certs\") pod \"db213002-5fde-4c7a-ba7e-6762c011b4a9\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " Nov 01 23:12:41.154757 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.154688 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-istio-envoy\") pod \"db213002-5fde-4c7a-ba7e-6762c011b4a9\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " Nov 01 23:12:41.154757 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.154719 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db213002-5fde-4c7a-ba7e-6762c011b4a9-kube-api-access-zpsqg\" (UniqueName: \"kubernetes.io/projected/db213002-5fde-4c7a-ba7e-6762c011b4a9-kube-api-access-zpsqg\") pod \"db213002-5fde-4c7a-ba7e-6762c011b4a9\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " Nov 01 23:12:41.155120 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.154762 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-credential-socket\") pod \"db213002-5fde-4c7a-ba7e-6762c011b4a9\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " Nov 01 23:12:41.155120 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.154795 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-istio-data" pod "db213002-5fde-4c7a-ba7e-6762c011b4a9" (UID: "db213002-5fde-4c7a-ba7e-6762c011b4a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:41.155120 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.154823 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db213002-5fde-4c7a-ba7e-6762c011b4a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db213002-5fde-4c7a-ba7e-6762c011b4a9-istio-podinfo\") pod \"db213002-5fde-4c7a-ba7e-6762c011b4a9\" (UID: \"db213002-5fde-4c7a-ba7e-6762c011b4a9\") " Nov 01 23:12:41.155120 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.154906 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-workload-socket" pod "db213002-5fde-4c7a-ba7e-6762c011b4a9" (UID: "db213002-5fde-4c7a-ba7e-6762c011b4a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:41.155338 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.155248 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.155338 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.155270 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.155444 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.155407 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-workload-certs" pod "db213002-5fde-4c7a-ba7e-6762c011b4a9" (UID: "db213002-5fde-4c7a-ba7e-6762c011b4a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:41.155516 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.155489 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-credential-socket" pod "db213002-5fde-4c7a-ba7e-6762c011b4a9" (UID: "db213002-5fde-4c7a-ba7e-6762c011b4a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:41.155965 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.155885 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db213002-5fde-4c7a-ba7e-6762c011b4a9-istiod-ca-cert" pod "db213002-5fde-4c7a-ba7e-6762c011b4a9" (UID: "db213002-5fde-4c7a-ba7e-6762c011b4a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:41.157537 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.157504 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-istio-envoy" pod "db213002-5fde-4c7a-ba7e-6762c011b4a9" (UID: "db213002-5fde-4c7a-ba7e-6762c011b4a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:41.157656 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.157544 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db213002-5fde-4c7a-ba7e-6762c011b4a9-istio-podinfo" pod "db213002-5fde-4c7a-ba7e-6762c011b4a9" (UID: "db213002-5fde-4c7a-ba7e-6762c011b4a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:12:41.157967 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.157716 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db213002-5fde-4c7a-ba7e-6762c011b4a9-istio-token" pod "db213002-5fde-4c7a-ba7e-6762c011b4a9" (UID: "db213002-5fde-4c7a-ba7e-6762c011b4a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:41.157967 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.157736 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db213002-5fde-4c7a-ba7e-6762c011b4a9-kube-api-access-zpsqg" pod "db213002-5fde-4c7a-ba7e-6762c011b4a9" (UID: "db213002-5fde-4c7a-ba7e-6762c011b4a9"). InnerVolumeSpecName "kube-api-access-zpsqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:41.256202 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.256117 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db213002-5fde-4c7a-ba7e-6762c011b4a9-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.256202 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.256151 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db213002-5fde-4c7a-ba7e-6762c011b4a9-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.256202 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.256167 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.256202 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.256178 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.256202 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.256191 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpsqg\" (UniqueName: \"kubernetes.io/projected/db213002-5fde-4c7a-ba7e-6762c011b4a9-kube-api-access-zpsqg\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.256532 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.256207 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db213002-5fde-4c7a-ba7e-6762c011b4a9-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.256532 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.256219 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db213002-5fde-4c7a-ba7e-6762c011b4a9-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.345105 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.345069 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz"] Nov 01 23:12:41.349437 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.349399 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-g4wcz"] Nov 01 23:12:41.749113 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.749074 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0d043d9-49a8-42fb-872c-cd5479590b1f" path="/var/lib/kubelet/pods/b0d043d9-49a8-42fb-872c-cd5479590b1f/volumes" Nov 01 23:12:41.749789 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:41.749767 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db213002-5fde-4c7a-ba7e-6762c011b4a9" path="/var/lib/kubelet/pods/db213002-5fde-4c7a-ba7e-6762c011b4a9/volumes" Nov 01 23:12:42.399289 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:42.399247 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-e5f47e-6b9bd944c5-jbtnp" podUID="fffb686c-2a8e-469c-bdd5-7eeb4b03bc06" containerName="payload-processing" containerID="cri-o://e544a092bee694d9400bd257e2d9a5b8a54bca2e2e1a488c37eeb46d9558f8dc" gracePeriod=25 Nov 01 23:12:42.692210 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:42.692187 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e5f47e-6b9bd944c5-jbtnp" Nov 01 23:12:42.774151 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:42.774104 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fffb686c-2a8e-469c-bdd5-7eeb4b03bc06-kube-api-access-dqblr\" (UniqueName: \"kubernetes.io/projected/fffb686c-2a8e-469c-bdd5-7eeb4b03bc06-kube-api-access-dqblr\") pod \"fffb686c-2a8e-469c-bdd5-7eeb4b03bc06\" (UID: \"fffb686c-2a8e-469c-bdd5-7eeb4b03bc06\") " Nov 01 23:12:42.774355 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:42.774175 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fffb686c-2a8e-469c-bdd5-7eeb4b03bc06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fffb686c-2a8e-469c-bdd5-7eeb4b03bc06-plugins-config-volume\") pod \"fffb686c-2a8e-469c-bdd5-7eeb4b03bc06\" (UID: \"fffb686c-2a8e-469c-bdd5-7eeb4b03bc06\") " Nov 01 23:12:42.774976 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:42.774943 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fffb686c-2a8e-469c-bdd5-7eeb4b03bc06-plugins-config-volume" pod "fffb686c-2a8e-469c-bdd5-7eeb4b03bc06" (UID: "fffb686c-2a8e-469c-bdd5-7eeb4b03bc06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:42.778022 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:42.777988 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fffb686c-2a8e-469c-bdd5-7eeb4b03bc06-kube-api-access-dqblr" pod "fffb686c-2a8e-469c-bdd5-7eeb4b03bc06" (UID: "fffb686c-2a8e-469c-bdd5-7eeb4b03bc06"). InnerVolumeSpecName "kube-api-access-dqblr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:42.875288 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:42.875253 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqblr\" (UniqueName: \"kubernetes.io/projected/fffb686c-2a8e-469c-bdd5-7eeb4b03bc06-kube-api-access-dqblr\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:42.875288 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:42.875283 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fffb686c-2a8e-469c-bdd5-7eeb4b03bc06-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:43.035364 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:43.035281 5011 generic.go:356] "Generic (PLEG): container finished" podID="fffb686c-2a8e-469c-bdd5-7eeb4b03bc06" containerID="e544a092bee694d9400bd257e2d9a5b8a54bca2e2e1a488c37eeb46d9558f8dc" exitCode=0 Nov 01 23:12:43.035516 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:43.035387 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e5f47e-6b9bd944c5-jbtnp" Nov 01 23:12:43.035516 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:43.035417 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e5f47e-6b9bd944c5-jbtnp" event={"ID":"fffb686c-2a8e-469c-bdd5-7eeb4b03bc06","Type":"ContainerDied","Data":"e544a092bee694d9400bd257e2d9a5b8a54bca2e2e1a488c37eeb46d9558f8dc"} Nov 01 23:12:43.035516 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:43.035485 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e5f47e-6b9bd944c5-jbtnp" event={"ID":"fffb686c-2a8e-469c-bdd5-7eeb4b03bc06","Type":"ContainerDied","Data":"09ddcebd51d852577956c92c6704bc6db6ce184e609581d6d891b3fa9cb5ebb1"} Nov 01 23:12:43.035673 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:43.035515 5011 scope.go:122] "RemoveContainer" containerID="e544a092bee694d9400bd257e2d9a5b8a54bca2e2e1a488c37eeb46d9558f8dc" Nov 01 23:12:43.049879 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:43.049852 5011 scope.go:122] "RemoveContainer" containerID="e544a092bee694d9400bd257e2d9a5b8a54bca2e2e1a488c37eeb46d9558f8dc" Nov 01 23:12:43.050258 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:12:43.050232 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e544a092bee694d9400bd257e2d9a5b8a54bca2e2e1a488c37eeb46d9558f8dc\": container with ID starting with e544a092bee694d9400bd257e2d9a5b8a54bca2e2e1a488c37eeb46d9558f8dc not found: ID does not exist" containerID="e544a092bee694d9400bd257e2d9a5b8a54bca2e2e1a488c37eeb46d9558f8dc" Nov 01 23:12:43.050382 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:43.050269 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e544a092bee694d9400bd257e2d9a5b8a54bca2e2e1a488c37eeb46d9558f8dc"} err="failed to get container status \"e544a092bee694d9400bd257e2d9a5b8a54bca2e2e1a488c37eeb46d9558f8dc\": rpc error: code = NotFound desc = could not find container \"e544a092bee694d9400bd257e2d9a5b8a54bca2e2e1a488c37eeb46d9558f8dc\": container with ID starting with e544a092bee694d9400bd257e2d9a5b8a54bca2e2e1a488c37eeb46d9558f8dc not found: ID does not exist" Nov 01 23:12:43.056823 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:43.056797 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e5f47e-6b9bd944c5-jbtnp"] Nov 01 23:12:43.060934 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:43.060913 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e5f47e-6b9bd944c5-jbtnp"] Nov 01 23:12:43.746265 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:43.746234 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fffb686c-2a8e-469c-bdd5-7eeb4b03bc06" path="/var/lib/kubelet/pods/fffb686c-2a8e-469c-bdd5-7eeb4b03bc06/volumes" Nov 01 23:12:46.959472 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:12:46.959434 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:12:47.107416 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.107374 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" containerID="cri-o://4f6b26b520e35f7be79e0cf120802d6d6dffbb1a8d003e21750e2a493aa8601e" gracePeriod=45 Nov 01 23:12:47.132318 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.132262 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" containerID="cri-o://1035e6b86781a2fb350ce93fda836ceb88d7316d442d678b9dabbf3e6ea56764" gracePeriod=45 Nov 01 23:12:47.388031 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.387992 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 23:12:47.424504 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.424461 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 23:12:47.518549 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.518461 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74441b26-2636-41c2-84e4-6a6be32d2793-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74441b26-2636-41c2-84e4-6a6be32d2793-model-cache\") pod \"74441b26-2636-41c2-84e4-6a6be32d2793\" (UID: \"74441b26-2636-41c2-84e4-6a6be32d2793\") " Nov 01 23:12:47.518549 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.518518 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74441b26-2636-41c2-84e4-6a6be32d2793-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74441b26-2636-41c2-84e4-6a6be32d2793-tmp-dir\") pod \"74441b26-2636-41c2-84e4-6a6be32d2793\" (UID: \"74441b26-2636-41c2-84e4-6a6be32d2793\") " Nov 01 23:12:47.518782 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.518555 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56eccabb-80c5-4796-bc93-f07a18bb160a-tls-certs\" (UniqueName: \"kubernetes.io/secret/56eccabb-80c5-4796-bc93-f07a18bb160a-tls-certs\") pod \"56eccabb-80c5-4796-bc93-f07a18bb160a\" (UID: \"56eccabb-80c5-4796-bc93-f07a18bb160a\") " Nov 01 23:12:47.518782 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.518574 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56eccabb-80c5-4796-bc93-f07a18bb160a-kube-api-access-sg2v5\" (UniqueName: \"kubernetes.io/projected/56eccabb-80c5-4796-bc93-f07a18bb160a-kube-api-access-sg2v5\") pod \"56eccabb-80c5-4796-bc93-f07a18bb160a\" (UID: \"56eccabb-80c5-4796-bc93-f07a18bb160a\") " Nov 01 23:12:47.518782 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.518639 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56eccabb-80c5-4796-bc93-f07a18bb160a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/56eccabb-80c5-4796-bc93-f07a18bb160a-dshm\") pod \"56eccabb-80c5-4796-bc93-f07a18bb160a\" (UID: \"56eccabb-80c5-4796-bc93-f07a18bb160a\") " Nov 01 23:12:47.518782 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.518694 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74441b26-2636-41c2-84e4-6a6be32d2793-tls-certs\" (UniqueName: \"kubernetes.io/secret/74441b26-2636-41c2-84e4-6a6be32d2793-tls-certs\") pod \"74441b26-2636-41c2-84e4-6a6be32d2793\" (UID: \"74441b26-2636-41c2-84e4-6a6be32d2793\") " Nov 01 23:12:47.518782 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.518764 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74441b26-2636-41c2-84e4-6a6be32d2793-model-cache" pod "74441b26-2636-41c2-84e4-6a6be32d2793" (UID: "74441b26-2636-41c2-84e4-6a6be32d2793"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:47.519087 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.518785 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56eccabb-80c5-4796-bc93-f07a18bb160a-home\" (UniqueName: \"kubernetes.io/empty-dir/56eccabb-80c5-4796-bc93-f07a18bb160a-home\") pod \"56eccabb-80c5-4796-bc93-f07a18bb160a\" (UID: \"56eccabb-80c5-4796-bc93-f07a18bb160a\") " Nov 01 23:12:47.519087 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.518815 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74441b26-2636-41c2-84e4-6a6be32d2793-tmp-dir" pod "74441b26-2636-41c2-84e4-6a6be32d2793" (UID: "74441b26-2636-41c2-84e4-6a6be32d2793"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:47.519087 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.518852 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74441b26-2636-41c2-84e4-6a6be32d2793-dshm\" (UniqueName: \"kubernetes.io/empty-dir/74441b26-2636-41c2-84e4-6a6be32d2793-dshm\") pod \"74441b26-2636-41c2-84e4-6a6be32d2793\" (UID: \"74441b26-2636-41c2-84e4-6a6be32d2793\") " Nov 01 23:12:47.519087 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.518918 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56eccabb-80c5-4796-bc93-f07a18bb160a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56eccabb-80c5-4796-bc93-f07a18bb160a-tmp-dir\") pod \"56eccabb-80c5-4796-bc93-f07a18bb160a\" (UID: \"56eccabb-80c5-4796-bc93-f07a18bb160a\") " Nov 01 23:12:47.519087 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.518965 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56eccabb-80c5-4796-bc93-f07a18bb160a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56eccabb-80c5-4796-bc93-f07a18bb160a-model-cache\") pod \"56eccabb-80c5-4796-bc93-f07a18bb160a\" (UID: \"56eccabb-80c5-4796-bc93-f07a18bb160a\") " Nov 01 23:12:47.519087 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.519003 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74441b26-2636-41c2-84e4-6a6be32d2793-kube-api-access-5zb8n\" (UniqueName: \"kubernetes.io/projected/74441b26-2636-41c2-84e4-6a6be32d2793-kube-api-access-5zb8n\") pod \"74441b26-2636-41c2-84e4-6a6be32d2793\" (UID: \"74441b26-2636-41c2-84e4-6a6be32d2793\") " Nov 01 23:12:47.519087 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.519035 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74441b26-2636-41c2-84e4-6a6be32d2793-home\" (UniqueName: \"kubernetes.io/empty-dir/74441b26-2636-41c2-84e4-6a6be32d2793-home\") pod \"74441b26-2636-41c2-84e4-6a6be32d2793\" (UID: \"74441b26-2636-41c2-84e4-6a6be32d2793\") " Nov 01 23:12:47.519624 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.519242 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56eccabb-80c5-4796-bc93-f07a18bb160a-home" pod "56eccabb-80c5-4796-bc93-f07a18bb160a" (UID: "56eccabb-80c5-4796-bc93-f07a18bb160a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:47.519624 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.519365 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56eccabb-80c5-4796-bc93-f07a18bb160a-model-cache" pod "56eccabb-80c5-4796-bc93-f07a18bb160a" (UID: "56eccabb-80c5-4796-bc93-f07a18bb160a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:47.519746 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.519631 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56eccabb-80c5-4796-bc93-f07a18bb160a-tmp-dir" pod "56eccabb-80c5-4796-bc93-f07a18bb160a" (UID: "56eccabb-80c5-4796-bc93-f07a18bb160a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:47.519987 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.519963 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74441b26-2636-41c2-84e4-6a6be32d2793-home" pod "74441b26-2636-41c2-84e4-6a6be32d2793" (UID: "74441b26-2636-41c2-84e4-6a6be32d2793"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:47.520310 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.520204 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56eccabb-80c5-4796-bc93-f07a18bb160a-home\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:47.520310 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.520230 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56eccabb-80c5-4796-bc93-f07a18bb160a-tmp-dir\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:47.520310 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.520245 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56eccabb-80c5-4796-bc93-f07a18bb160a-model-cache\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:47.520310 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.520259 5011 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74441b26-2636-41c2-84e4-6a6be32d2793-home\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:47.520310 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.520273 5011 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74441b26-2636-41c2-84e4-6a6be32d2793-model-cache\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:47.520310 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.520286 5011 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74441b26-2636-41c2-84e4-6a6be32d2793-tmp-dir\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:47.521452 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.521431 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74441b26-2636-41c2-84e4-6a6be32d2793-dshm" pod "74441b26-2636-41c2-84e4-6a6be32d2793" (UID: "74441b26-2636-41c2-84e4-6a6be32d2793"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:47.521711 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.521675 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56eccabb-80c5-4796-bc93-f07a18bb160a-tls-certs" pod "56eccabb-80c5-4796-bc93-f07a18bb160a" (UID: "56eccabb-80c5-4796-bc93-f07a18bb160a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:12:47.521711 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.521678 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56eccabb-80c5-4796-bc93-f07a18bb160a-dshm" pod "56eccabb-80c5-4796-bc93-f07a18bb160a" (UID: "56eccabb-80c5-4796-bc93-f07a18bb160a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:47.521711 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.521691 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74441b26-2636-41c2-84e4-6a6be32d2793-tls-certs" pod "74441b26-2636-41c2-84e4-6a6be32d2793" (UID: "74441b26-2636-41c2-84e4-6a6be32d2793"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:12:47.522329 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.522306 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74441b26-2636-41c2-84e4-6a6be32d2793-kube-api-access-5zb8n" pod "74441b26-2636-41c2-84e4-6a6be32d2793" (UID: "74441b26-2636-41c2-84e4-6a6be32d2793"). InnerVolumeSpecName "kube-api-access-5zb8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:47.522400 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.522352 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56eccabb-80c5-4796-bc93-f07a18bb160a-kube-api-access-sg2v5" pod "56eccabb-80c5-4796-bc93-f07a18bb160a" (UID: "56eccabb-80c5-4796-bc93-f07a18bb160a"). InnerVolumeSpecName "kube-api-access-sg2v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:47.620799 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.620765 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56eccabb-80c5-4796-bc93-f07a18bb160a-tls-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:47.620799 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.620798 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg2v5\" (UniqueName: \"kubernetes.io/projected/56eccabb-80c5-4796-bc93-f07a18bb160a-kube-api-access-sg2v5\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:47.621044 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.620813 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56eccabb-80c5-4796-bc93-f07a18bb160a-dshm\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:47.621044 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.620826 5011 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74441b26-2636-41c2-84e4-6a6be32d2793-tls-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:47.621044 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.620864 5011 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74441b26-2636-41c2-84e4-6a6be32d2793-dshm\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:47.621044 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:47.620877 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zb8n\" (UniqueName: \"kubernetes.io/projected/74441b26-2636-41c2-84e4-6a6be32d2793-kube-api-access-5zb8n\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:48.062957 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:48.062923 5011 generic.go:356] "Generic (PLEG): container finished" podID="74441b26-2636-41c2-84e4-6a6be32d2793" containerID="1035e6b86781a2fb350ce93fda836ceb88d7316d442d678b9dabbf3e6ea56764" exitCode=0 Nov 01 23:12:48.063518 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:48.062971 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" Nov 01 23:12:48.063518 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:48.063007 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" event={"ID":"74441b26-2636-41c2-84e4-6a6be32d2793","Type":"ContainerDied","Data":"1035e6b86781a2fb350ce93fda836ceb88d7316d442d678b9dabbf3e6ea56764"} Nov 01 23:12:48.063518 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:48.063050 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr" event={"ID":"74441b26-2636-41c2-84e4-6a6be32d2793","Type":"ContainerDied","Data":"a75cec771050da40bb02b0229efb9f53d2b0da64502c1d14e27cb20b6b8f77e5"} Nov 01 23:12:48.063518 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:48.063067 5011 scope.go:122] "RemoveContainer" containerID="1035e6b86781a2fb350ce93fda836ceb88d7316d442d678b9dabbf3e6ea56764" Nov 01 23:12:48.064452 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:48.064428 5011 generic.go:356] "Generic (PLEG): container finished" podID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerID="4f6b26b520e35f7be79e0cf120802d6d6dffbb1a8d003e21750e2a493aa8601e" exitCode=0 Nov 01 23:12:48.064563 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:48.064467 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" Nov 01 23:12:48.064563 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:48.064465 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" event={"ID":"56eccabb-80c5-4796-bc93-f07a18bb160a","Type":"ContainerDied","Data":"4f6b26b520e35f7be79e0cf120802d6d6dffbb1a8d003e21750e2a493aa8601e"} Nov 01 23:12:48.064563 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:48.064519 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t" event={"ID":"56eccabb-80c5-4796-bc93-f07a18bb160a","Type":"ContainerDied","Data":"ce1106520e16408ce876d1df8eba3d89f3d8f40031cb5adc12982217d1cd912c"} Nov 01 23:12:48.072595 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:48.072579 5011 scope.go:122] "RemoveContainer" containerID="1035e6b86781a2fb350ce93fda836ceb88d7316d442d678b9dabbf3e6ea56764" Nov 01 23:12:48.072907 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:12:48.072877 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1035e6b86781a2fb350ce93fda836ceb88d7316d442d678b9dabbf3e6ea56764\": container with ID starting with 1035e6b86781a2fb350ce93fda836ceb88d7316d442d678b9dabbf3e6ea56764 not found: ID does not exist" containerID="1035e6b86781a2fb350ce93fda836ceb88d7316d442d678b9dabbf3e6ea56764" Nov 01 23:12:48.073032 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:48.072915 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1035e6b86781a2fb350ce93fda836ceb88d7316d442d678b9dabbf3e6ea56764"} err="failed to get container status \"1035e6b86781a2fb350ce93fda836ceb88d7316d442d678b9dabbf3e6ea56764\": rpc error: code = NotFound desc = could not find container \"1035e6b86781a2fb350ce93fda836ceb88d7316d442d678b9dabbf3e6ea56764\": container with ID starting with 1035e6b86781a2fb350ce93fda836ceb88d7316d442d678b9dabbf3e6ea56764 not found: ID does not exist" Nov 01 23:12:48.073032 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:48.072941 5011 scope.go:122] "RemoveContainer" containerID="4f6b26b520e35f7be79e0cf120802d6d6dffbb1a8d003e21750e2a493aa8601e" Nov 01 23:12:48.079796 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:48.079774 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr"] Nov 01 23:12:48.081596 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:48.081557 5011 scope.go:122] "RemoveContainer" containerID="4f6b26b520e35f7be79e0cf120802d6d6dffbb1a8d003e21750e2a493aa8601e" Nov 01 23:12:48.081861 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:12:48.081819 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f6b26b520e35f7be79e0cf120802d6d6dffbb1a8d003e21750e2a493aa8601e\": container with ID starting with 4f6b26b520e35f7be79e0cf120802d6d6dffbb1a8d003e21750e2a493aa8601e not found: ID does not exist" containerID="4f6b26b520e35f7be79e0cf120802d6d6dffbb1a8d003e21750e2a493aa8601e" Nov 01 23:12:48.081977 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:48.081874 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f6b26b520e35f7be79e0cf120802d6d6dffbb1a8d003e21750e2a493aa8601e"} err="failed to get container status \"4f6b26b520e35f7be79e0cf120802d6d6dffbb1a8d003e21750e2a493aa8601e\": rpc error: code = NotFound desc = could not find container \"4f6b26b520e35f7be79e0cf120802d6d6dffbb1a8d003e21750e2a493aa8601e\": container with ID starting with 4f6b26b520e35f7be79e0cf120802d6d6dffbb1a8d003e21750e2a493aa8601e not found: ID does not exist" Nov 01 23:12:48.084228 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:48.084206 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e5f47e/simulator-w0-e5f47e-kserve-cb9b94dc9-fnlxr"] Nov 01 23:12:48.094953 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:48.094932 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t"] Nov 01 23:12:48.100893 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:48.100870 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e5f47e/distinct-2-w0-e5f47e-kserve-664ddc4d5d-lwd2t"] Nov 01 23:12:49.745248 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:49.745214 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" path="/var/lib/kubelet/pods/56eccabb-80c5-4796-bc93-f07a18bb160a/volumes" Nov 01 23:12:49.745692 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:49.745677 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" path="/var/lib/kubelet/pods/74441b26-2636-41c2-84e4-6a6be32d2793/volumes" Nov 01 23:12:49.796017 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:49.795984 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:12:52.596121 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:52.596083 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk"] Nov 01 23:12:52.596684 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:52.596598 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk" podUID="abea67e1-7f6b-4abd-a1f1-13a2b1613458" containerName="maas-api" containerID="cri-o://7af8b71d05d028efb8b8aae590fe687c5e12cd5fc70e25162ff94a40d45b95d0" gracePeriod=30 Nov 01 23:12:52.853882 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:52.853797 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk" Nov 01 23:12:52.870577 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:52.870549 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abea67e1-7f6b-4abd-a1f1-13a2b1613458-metrics-certs\" (UniqueName: \"kubernetes.io/secret/abea67e1-7f6b-4abd-a1f1-13a2b1613458-metrics-certs\") pod \"abea67e1-7f6b-4abd-a1f1-13a2b1613458\" (UID: \"abea67e1-7f6b-4abd-a1f1-13a2b1613458\") " Nov 01 23:12:52.870720 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:52.870596 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abea67e1-7f6b-4abd-a1f1-13a2b1613458-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abea67e1-7f6b-4abd-a1f1-13a2b1613458-maas-api-tls\") pod \"abea67e1-7f6b-4abd-a1f1-13a2b1613458\" (UID: \"abea67e1-7f6b-4abd-a1f1-13a2b1613458\") " Nov 01 23:12:52.870720 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:52.870704 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abea67e1-7f6b-4abd-a1f1-13a2b1613458-kube-api-access-5d2zs\" (UniqueName: \"kubernetes.io/projected/abea67e1-7f6b-4abd-a1f1-13a2b1613458-kube-api-access-5d2zs\") pod \"abea67e1-7f6b-4abd-a1f1-13a2b1613458\" (UID: \"abea67e1-7f6b-4abd-a1f1-13a2b1613458\") " Nov 01 23:12:52.870850 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:52.870818 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abea67e1-7f6b-4abd-a1f1-13a2b1613458-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abea67e1-7f6b-4abd-a1f1-13a2b1613458-openshift-service-ca\") pod \"abea67e1-7f6b-4abd-a1f1-13a2b1613458\" (UID: \"abea67e1-7f6b-4abd-a1f1-13a2b1613458\") " Nov 01 23:12:52.871148 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:52.871119 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abea67e1-7f6b-4abd-a1f1-13a2b1613458-openshift-service-ca" pod "abea67e1-7f6b-4abd-a1f1-13a2b1613458" (UID: "abea67e1-7f6b-4abd-a1f1-13a2b1613458"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:52.872972 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:52.872940 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abea67e1-7f6b-4abd-a1f1-13a2b1613458-kube-api-access-5d2zs" pod "abea67e1-7f6b-4abd-a1f1-13a2b1613458" (UID: "abea67e1-7f6b-4abd-a1f1-13a2b1613458"). InnerVolumeSpecName "kube-api-access-5d2zs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:52.873096 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:52.872981 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abea67e1-7f6b-4abd-a1f1-13a2b1613458-metrics-certs" pod "abea67e1-7f6b-4abd-a1f1-13a2b1613458" (UID: "abea67e1-7f6b-4abd-a1f1-13a2b1613458"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:12:52.873096 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:52.872994 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abea67e1-7f6b-4abd-a1f1-13a2b1613458-maas-api-tls" pod "abea67e1-7f6b-4abd-a1f1-13a2b1613458" (UID: "abea67e1-7f6b-4abd-a1f1-13a2b1613458"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:12:52.972238 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:52.972200 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d2zs\" (UniqueName: \"kubernetes.io/projected/abea67e1-7f6b-4abd-a1f1-13a2b1613458-kube-api-access-5d2zs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:52.972238 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:52.972237 5011 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abea67e1-7f6b-4abd-a1f1-13a2b1613458-openshift-service-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:52.972452 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:52.972253 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abea67e1-7f6b-4abd-a1f1-13a2b1613458-metrics-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:52.972452 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:52.972269 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abea67e1-7f6b-4abd-a1f1-13a2b1613458-maas-api-tls\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:53.094605 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:53.094570 5011 generic.go:356] "Generic (PLEG): container finished" podID="abea67e1-7f6b-4abd-a1f1-13a2b1613458" containerID="7af8b71d05d028efb8b8aae590fe687c5e12cd5fc70e25162ff94a40d45b95d0" exitCode=0 Nov 01 23:12:53.094778 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:53.094616 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk" Nov 01 23:12:53.094778 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:53.094657 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk" event={"ID":"abea67e1-7f6b-4abd-a1f1-13a2b1613458","Type":"ContainerDied","Data":"7af8b71d05d028efb8b8aae590fe687c5e12cd5fc70e25162ff94a40d45b95d0"} Nov 01 23:12:53.094778 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:53.094715 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk" event={"ID":"abea67e1-7f6b-4abd-a1f1-13a2b1613458","Type":"ContainerDied","Data":"18f4fa6a44e12e0bbf1fbf1465246362e55354474f36832eb3030668e3e31331"} Nov 01 23:12:53.094778 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:53.094739 5011 scope.go:122] "RemoveContainer" containerID="7af8b71d05d028efb8b8aae590fe687c5e12cd5fc70e25162ff94a40d45b95d0" Nov 01 23:12:53.106195 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:53.106174 5011 scope.go:122] "RemoveContainer" containerID="7af8b71d05d028efb8b8aae590fe687c5e12cd5fc70e25162ff94a40d45b95d0" Nov 01 23:12:53.106556 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:12:53.106529 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7af8b71d05d028efb8b8aae590fe687c5e12cd5fc70e25162ff94a40d45b95d0\": container with ID starting with 7af8b71d05d028efb8b8aae590fe687c5e12cd5fc70e25162ff94a40d45b95d0 not found: ID does not exist" containerID="7af8b71d05d028efb8b8aae590fe687c5e12cd5fc70e25162ff94a40d45b95d0" Nov 01 23:12:53.106642 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:53.106569 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7af8b71d05d028efb8b8aae590fe687c5e12cd5fc70e25162ff94a40d45b95d0"} err="failed to get container status \"7af8b71d05d028efb8b8aae590fe687c5e12cd5fc70e25162ff94a40d45b95d0\": rpc error: code = NotFound desc = could not find container \"7af8b71d05d028efb8b8aae590fe687c5e12cd5fc70e25162ff94a40d45b95d0\": container with ID starting with 7af8b71d05d028efb8b8aae590fe687c5e12cd5fc70e25162ff94a40d45b95d0 not found: ID does not exist" Nov 01 23:12:53.128171 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:53.128133 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk"] Nov 01 23:12:53.140361 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:53.140332 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-e5f47e-d99d89db5-xt8lk"] Nov 01 23:12:53.745098 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:53.745066 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abea67e1-7f6b-4abd-a1f1-13a2b1613458" path="/var/lib/kubelet/pods/abea67e1-7f6b-4abd-a1f1-13a2b1613458/volumes" Nov 01 23:12:55.375571 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:55.375539 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:12:55.525861 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:55.524573 5011 scope.go:122] "RemoveContainer" containerID="c3611b76aac7926b7ff46797106c1d74fcfac16be77ac022e876f5ff6c4bc880" Nov 01 23:12:59.557156 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:59.557115 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml"] Nov 01 23:12:59.557694 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:12:59.557369 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" podUID="a29727f6-5841-404b-aa67-d596290c08d1" containerName="istio-proxy" containerID="cri-o://061199c7094d1a5c86e7c1de635a87beb8ed2ae6a2eb6886736e8d8f11179152" gracePeriod=30 Nov 01 23:13:02.062365 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:02.062324 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66565fcd54-xfn5s"] Nov 01 23:13:02.062951 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:02.062627 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-66565fcd54-xfn5s" podUID="4f6e51ef-8558-4cc7-9c47-77dba02aadc8" containerName="manager" containerID="cri-o://60547e47f7252507a1a21ea8fe8a32eb170a3859bc0818ddb93d0301cae5ec8b" gracePeriod=30 Nov 01 23:13:02.310350 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:02.310327 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66565fcd54-xfn5s" Nov 01 23:13:02.363563 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:02.363475 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f6e51ef-8558-4cc7-9c47-77dba02aadc8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f6e51ef-8558-4cc7-9c47-77dba02aadc8-metrics-certs\") pod \"4f6e51ef-8558-4cc7-9c47-77dba02aadc8\" (UID: \"4f6e51ef-8558-4cc7-9c47-77dba02aadc8\") " Nov 01 23:13:02.363713 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:02.363663 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f6e51ef-8558-4cc7-9c47-77dba02aadc8-webhook-cert\" (UniqueName: \"kubernetes.io/secret/4f6e51ef-8558-4cc7-9c47-77dba02aadc8-webhook-cert\") pod \"4f6e51ef-8558-4cc7-9c47-77dba02aadc8\" (UID: \"4f6e51ef-8558-4cc7-9c47-77dba02aadc8\") " Nov 01 23:13:02.363713 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:02.363699 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f6e51ef-8558-4cc7-9c47-77dba02aadc8-kube-api-access-z8b5k\" (UniqueName: \"kubernetes.io/projected/4f6e51ef-8558-4cc7-9c47-77dba02aadc8-kube-api-access-z8b5k\") pod \"4f6e51ef-8558-4cc7-9c47-77dba02aadc8\" (UID: \"4f6e51ef-8558-4cc7-9c47-77dba02aadc8\") " Nov 01 23:13:02.365863 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:02.365779 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f6e51ef-8558-4cc7-9c47-77dba02aadc8-webhook-cert" pod "4f6e51ef-8558-4cc7-9c47-77dba02aadc8" (UID: "4f6e51ef-8558-4cc7-9c47-77dba02aadc8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:13:02.366004 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:02.365897 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f6e51ef-8558-4cc7-9c47-77dba02aadc8-metrics-certs" pod "4f6e51ef-8558-4cc7-9c47-77dba02aadc8" (UID: "4f6e51ef-8558-4cc7-9c47-77dba02aadc8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:13:02.366192 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:02.366168 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f6e51ef-8558-4cc7-9c47-77dba02aadc8-kube-api-access-z8b5k" pod "4f6e51ef-8558-4cc7-9c47-77dba02aadc8" (UID: "4f6e51ef-8558-4cc7-9c47-77dba02aadc8"). InnerVolumeSpecName "kube-api-access-z8b5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:13:02.465202 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:02.465165 5011 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4f6e51ef-8558-4cc7-9c47-77dba02aadc8-webhook-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:13:02.465202 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:02.465197 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8b5k\" (UniqueName: \"kubernetes.io/projected/4f6e51ef-8558-4cc7-9c47-77dba02aadc8-kube-api-access-z8b5k\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:13:02.465202 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:02.465207 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f6e51ef-8558-4cc7-9c47-77dba02aadc8-metrics-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:13:03.143470 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.143430 5011 generic.go:356] "Generic (PLEG): container finished" podID="4f6e51ef-8558-4cc7-9c47-77dba02aadc8" containerID="60547e47f7252507a1a21ea8fe8a32eb170a3859bc0818ddb93d0301cae5ec8b" exitCode=0 Nov 01 23:13:03.144008 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.143477 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66565fcd54-xfn5s" Nov 01 23:13:03.144008 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.143522 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66565fcd54-xfn5s" event={"ID":"4f6e51ef-8558-4cc7-9c47-77dba02aadc8","Type":"ContainerDied","Data":"60547e47f7252507a1a21ea8fe8a32eb170a3859bc0818ddb93d0301cae5ec8b"} Nov 01 23:13:03.144008 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.143582 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66565fcd54-xfn5s" event={"ID":"4f6e51ef-8558-4cc7-9c47-77dba02aadc8","Type":"ContainerDied","Data":"a5dcd2d55443fc4ea8cd29c4dd081f2f4337d105a2c9d79486bfcf7689803f90"} Nov 01 23:13:03.144008 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.143612 5011 scope.go:122] "RemoveContainer" containerID="60547e47f7252507a1a21ea8fe8a32eb170a3859bc0818ddb93d0301cae5ec8b" Nov 01 23:13:03.153885 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.153860 5011 scope.go:122] "RemoveContainer" containerID="60547e47f7252507a1a21ea8fe8a32eb170a3859bc0818ddb93d0301cae5ec8b" Nov 01 23:13:03.154176 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:13:03.154157 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60547e47f7252507a1a21ea8fe8a32eb170a3859bc0818ddb93d0301cae5ec8b\": container with ID starting with 60547e47f7252507a1a21ea8fe8a32eb170a3859bc0818ddb93d0301cae5ec8b not found: ID does not exist" containerID="60547e47f7252507a1a21ea8fe8a32eb170a3859bc0818ddb93d0301cae5ec8b" Nov 01 23:13:03.154260 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.154190 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60547e47f7252507a1a21ea8fe8a32eb170a3859bc0818ddb93d0301cae5ec8b"} err="failed to get container status \"60547e47f7252507a1a21ea8fe8a32eb170a3859bc0818ddb93d0301cae5ec8b\": rpc error: code = NotFound desc = could not find container \"60547e47f7252507a1a21ea8fe8a32eb170a3859bc0818ddb93d0301cae5ec8b\": container with ID starting with 60547e47f7252507a1a21ea8fe8a32eb170a3859bc0818ddb93d0301cae5ec8b not found: ID does not exist" Nov 01 23:13:03.165290 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.165065 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66565fcd54-xfn5s"] Nov 01 23:13:03.167020 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.166969 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-66565fcd54-xfn5s"] Nov 01 23:13:03.745682 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.745643 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f6e51ef-8558-4cc7-9c47-77dba02aadc8" path="/var/lib/kubelet/pods/4f6e51ef-8558-4cc7-9c47-77dba02aadc8/volumes" Nov 01 23:13:03.795679 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.795645 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-66565fcd54-x4c6t"] Nov 01 23:13:03.796130 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.796112 5011 cpu_manager.go:414] "containerMap: removing container" podUID="b0d043d9-49a8-42fb-872c-cd5479590b1f" containerName="istio-proxy" Nov 01 23:13:03.796244 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.796132 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0d043d9-49a8-42fb-872c-cd5479590b1f" containerName="istio-proxy" Nov 01 23:13:03.796244 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.796153 5011 cpu_manager.go:414] "containerMap: removing container" podUID="fffb686c-2a8e-469c-bdd5-7eeb4b03bc06" containerName="payload-processing" Nov 01 23:13:03.796244 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.796161 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fffb686c-2a8e-469c-bdd5-7eeb4b03bc06" containerName="payload-processing" Nov 01 23:13:03.796244 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.796182 5011 cpu_manager.go:414] "containerMap: removing container" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" Nov 01 23:13:03.796244 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.796191 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" Nov 01 23:13:03.796244 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.796204 5011 cpu_manager.go:414] "containerMap: removing container" podUID="4f6e51ef-8558-4cc7-9c47-77dba02aadc8" containerName="manager" Nov 01 23:13:03.796244 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.796211 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f6e51ef-8558-4cc7-9c47-77dba02aadc8" containerName="manager" Nov 01 23:13:03.796244 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.796227 5011 cpu_manager.go:414] "containerMap: removing container" podUID="80493f88-4b1b-4d2c-acb5-f956a719af86" containerName="revoke-keys" Nov 01 23:13:03.796663 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.796249 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80493f88-4b1b-4d2c-acb5-f956a719af86" containerName="revoke-keys" Nov 01 23:13:03.796663 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.796262 5011 cpu_manager.go:414] "containerMap: removing container" podUID="db213002-5fde-4c7a-ba7e-6762c011b4a9" containerName="istio-proxy" Nov 01 23:13:03.796663 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.796271 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db213002-5fde-4c7a-ba7e-6762c011b4a9" containerName="istio-proxy" Nov 01 23:13:03.796663 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.796285 5011 cpu_manager.go:414] "containerMap: removing container" podUID="abea67e1-7f6b-4abd-a1f1-13a2b1613458" containerName="maas-api" Nov 01 23:13:03.796663 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.796292 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abea67e1-7f6b-4abd-a1f1-13a2b1613458" containerName="maas-api" Nov 01 23:13:03.796663 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.796302 5011 cpu_manager.go:414] "containerMap: removing container" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" Nov 01 23:13:03.796663 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.796309 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" Nov 01 23:13:03.796663 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.796410 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0d043d9-49a8-42fb-872c-cd5479590b1f" containerName="istio-proxy" Nov 01 23:13:03.796663 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.796425 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="fffb686c-2a8e-469c-bdd5-7eeb4b03bc06" containerName="payload-processing" Nov 01 23:13:03.796663 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.796437 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="74441b26-2636-41c2-84e4-6a6be32d2793" containerName="main" Nov 01 23:13:03.796663 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.796446 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f6e51ef-8558-4cc7-9c47-77dba02aadc8" containerName="manager" Nov 01 23:13:03.796663 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.796456 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="80493f88-4b1b-4d2c-acb5-f956a719af86" containerName="revoke-keys" Nov 01 23:13:03.796663 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.796464 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="db213002-5fde-4c7a-ba7e-6762c011b4a9" containerName="istio-proxy" Nov 01 23:13:03.796663 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.796473 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="abea67e1-7f6b-4abd-a1f1-13a2b1613458" containerName="maas-api" Nov 01 23:13:03.796663 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.796481 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="56eccabb-80c5-4796-bc93-f07a18bb160a" containerName="main" Nov 01 23:13:03.801260 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.801239 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66565fcd54-x4c6t" Nov 01 23:13:03.806534 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.806508 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 01 23:13:03.810308 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.810281 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66565fcd54-x4c6t"] Nov 01 23:13:03.810442 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.810397 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 01 23:13:03.810442 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.810417 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ztmdx\"" Nov 01 23:13:03.810582 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.810455 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 01 23:13:03.885475 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.885440 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f4631edc-4126-4c12-85a6-8286a976d741-webhook-cert\") pod \"maas-controller-66565fcd54-x4c6t\" (UID: \"f4631edc-4126-4c12-85a6-8286a976d741\") " pod="opendatahub/maas-controller-66565fcd54-x4c6t" Nov 01 23:13:03.885655 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.885508 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cjtz\" (UniqueName: \"kubernetes.io/projected/f4631edc-4126-4c12-85a6-8286a976d741-kube-api-access-7cjtz\") pod \"maas-controller-66565fcd54-x4c6t\" (UID: \"f4631edc-4126-4c12-85a6-8286a976d741\") " pod="opendatahub/maas-controller-66565fcd54-x4c6t" Nov 01 23:13:03.885655 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.885584 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4631edc-4126-4c12-85a6-8286a976d741-metrics-certs\") pod \"maas-controller-66565fcd54-x4c6t\" (UID: \"f4631edc-4126-4c12-85a6-8286a976d741\") " pod="opendatahub/maas-controller-66565fcd54-x4c6t" Nov 01 23:13:03.986311 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.986279 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cjtz\" (UniqueName: \"kubernetes.io/projected/f4631edc-4126-4c12-85a6-8286a976d741-kube-api-access-7cjtz\") pod \"maas-controller-66565fcd54-x4c6t\" (UID: \"f4631edc-4126-4c12-85a6-8286a976d741\") " pod="opendatahub/maas-controller-66565fcd54-x4c6t" Nov 01 23:13:03.986490 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.986327 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4631edc-4126-4c12-85a6-8286a976d741-metrics-certs\") pod \"maas-controller-66565fcd54-x4c6t\" (UID: \"f4631edc-4126-4c12-85a6-8286a976d741\") " pod="opendatahub/maas-controller-66565fcd54-x4c6t" Nov 01 23:13:03.986490 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.986381 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f4631edc-4126-4c12-85a6-8286a976d741-webhook-cert\") pod \"maas-controller-66565fcd54-x4c6t\" (UID: \"f4631edc-4126-4c12-85a6-8286a976d741\") " pod="opendatahub/maas-controller-66565fcd54-x4c6t" Nov 01 23:13:03.989058 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.989029 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f4631edc-4126-4c12-85a6-8286a976d741-webhook-cert\") pod \"maas-controller-66565fcd54-x4c6t\" (UID: \"f4631edc-4126-4c12-85a6-8286a976d741\") " pod="opendatahub/maas-controller-66565fcd54-x4c6t" Nov 01 23:13:03.989058 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.989051 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4631edc-4126-4c12-85a6-8286a976d741-metrics-certs\") pod \"maas-controller-66565fcd54-x4c6t\" (UID: \"f4631edc-4126-4c12-85a6-8286a976d741\") " pod="opendatahub/maas-controller-66565fcd54-x4c6t" Nov 01 23:13:03.995101 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:03.995076 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cjtz\" (UniqueName: \"kubernetes.io/projected/f4631edc-4126-4c12-85a6-8286a976d741-kube-api-access-7cjtz\") pod \"maas-controller-66565fcd54-x4c6t\" (UID: \"f4631edc-4126-4c12-85a6-8286a976d741\") " pod="opendatahub/maas-controller-66565fcd54-x4c6t" Nov 01 23:13:04.118429 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:04.118334 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66565fcd54-x4c6t" Nov 01 23:13:04.399694 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:04.399665 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66565fcd54-x4c6t"] Nov 01 23:13:04.401819 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:13:04.401778 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4631edc_4126_4c12_85a6_8286a976d741.slice/crio-fed856f6871116c67d0371b5196d3bed1383dc967957f6f341866ce788338c0e WatchSource:0}: Error finding container fed856f6871116c67d0371b5196d3bed1383dc967957f6f341866ce788338c0e: Status 404 returned error can't find the container with id fed856f6871116c67d0371b5196d3bed1383dc967957f6f341866ce788338c0e Nov 01 23:13:04.403931 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:04.403905 5011 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 23:13:04.818447 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:04.818423 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 23:13:04.898088 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:04.898033 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-credential-socket\") pod \"a29727f6-5841-404b-aa67-d596290c08d1\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " Nov 01 23:13:04.898088 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:04.898091 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-istio-envoy\") pod \"a29727f6-5841-404b-aa67-d596290c08d1\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " Nov 01 23:13:04.898362 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:04.898143 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-workload-certs\") pod \"a29727f6-5841-404b-aa67-d596290c08d1\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " Nov 01 23:13:04.898362 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:04.898204 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a29727f6-5841-404b-aa67-d596290c08d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a29727f6-5841-404b-aa67-d596290c08d1-istiod-ca-cert\") pod \"a29727f6-5841-404b-aa67-d596290c08d1\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " Nov 01 23:13:04.898362 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:04.898243 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29727f6-5841-404b-aa67-d596290c08d1-istio-token\" (UniqueName: \"kubernetes.io/projected/a29727f6-5841-404b-aa67-d596290c08d1-istio-token\") pod \"a29727f6-5841-404b-aa67-d596290c08d1\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " Nov 01 23:13:04.898362 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:04.898306 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-workload-socket\") pod \"a29727f6-5841-404b-aa67-d596290c08d1\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " Nov 01 23:13:04.898631 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:04.898376 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29727f6-5841-404b-aa67-d596290c08d1-kube-api-access-g8qnb\" (UniqueName: \"kubernetes.io/projected/a29727f6-5841-404b-aa67-d596290c08d1-kube-api-access-g8qnb\") pod \"a29727f6-5841-404b-aa67-d596290c08d1\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " Nov 01 23:13:04.898631 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:04.898385 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-credential-socket" pod "a29727f6-5841-404b-aa67-d596290c08d1" (UID: "a29727f6-5841-404b-aa67-d596290c08d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:13:04.898631 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:04.898420 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a29727f6-5841-404b-aa67-d596290c08d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a29727f6-5841-404b-aa67-d596290c08d1-istio-podinfo\") pod \"a29727f6-5841-404b-aa67-d596290c08d1\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " Nov 01 23:13:04.898631 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:04.898487 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-istio-data\") pod \"a29727f6-5841-404b-aa67-d596290c08d1\" (UID: \"a29727f6-5841-404b-aa67-d596290c08d1\") " Nov 01 23:13:04.898938 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:04.898816 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:13:04.899223 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:04.899060 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-istio-data" pod "a29727f6-5841-404b-aa67-d596290c08d1" (UID: "a29727f6-5841-404b-aa67-d596290c08d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:13:04.899223 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:04.899168 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-workload-socket" pod "a29727f6-5841-404b-aa67-d596290c08d1" (UID: "a29727f6-5841-404b-aa67-d596290c08d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:13:04.899223 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:04.899171 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-workload-certs" pod "a29727f6-5841-404b-aa67-d596290c08d1" (UID: "a29727f6-5841-404b-aa67-d596290c08d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:13:04.899476 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:04.899337 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29727f6-5841-404b-aa67-d596290c08d1-istiod-ca-cert" pod "a29727f6-5841-404b-aa67-d596290c08d1" (UID: "a29727f6-5841-404b-aa67-d596290c08d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:13:04.901382 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:04.901353 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29727f6-5841-404b-aa67-d596290c08d1-istio-token" pod "a29727f6-5841-404b-aa67-d596290c08d1" (UID: "a29727f6-5841-404b-aa67-d596290c08d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:13:04.901382 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:04.901369 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-istio-envoy" pod "a29727f6-5841-404b-aa67-d596290c08d1" (UID: "a29727f6-5841-404b-aa67-d596290c08d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:13:04.901516 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:04.901391 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29727f6-5841-404b-aa67-d596290c08d1-kube-api-access-g8qnb" pod "a29727f6-5841-404b-aa67-d596290c08d1" (UID: "a29727f6-5841-404b-aa67-d596290c08d1"). InnerVolumeSpecName "kube-api-access-g8qnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:13:04.901886 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:04.901864 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a29727f6-5841-404b-aa67-d596290c08d1-istio-podinfo" pod "a29727f6-5841-404b-aa67-d596290c08d1" (UID: "a29727f6-5841-404b-aa67-d596290c08d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:13:05.000015 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:04.999979 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a29727f6-5841-404b-aa67-d596290c08d1-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:13:05.000015 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:05.000010 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a29727f6-5841-404b-aa67-d596290c08d1-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:13:05.000250 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:05.000026 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:13:05.000250 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:05.000041 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8qnb\" (UniqueName: \"kubernetes.io/projected/a29727f6-5841-404b-aa67-d596290c08d1-kube-api-access-g8qnb\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:13:05.000250 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:05.000057 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a29727f6-5841-404b-aa67-d596290c08d1-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:13:05.000250 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:05.000070 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:13:05.000250 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:05.000083 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:13:05.000250 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:05.000098 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a29727f6-5841-404b-aa67-d596290c08d1-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:13:05.159263 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:05.159225 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66565fcd54-x4c6t" event={"ID":"f4631edc-4126-4c12-85a6-8286a976d741","Type":"ContainerStarted","Data":"85414a9e94843a9e8eab7e5b74b620dde38ab7be0200e687df4a9307b6ee271e"} Nov 01 23:13:05.159446 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:05.159296 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66565fcd54-x4c6t" event={"ID":"f4631edc-4126-4c12-85a6-8286a976d741","Type":"ContainerStarted","Data":"fed856f6871116c67d0371b5196d3bed1383dc967957f6f341866ce788338c0e"} Nov 01 23:13:05.159446 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:05.159345 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-66565fcd54-x4c6t" Nov 01 23:13:05.161041 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:05.161006 5011 generic.go:356] "Generic (PLEG): container finished" podID="a29727f6-5841-404b-aa67-d596290c08d1" containerID="061199c7094d1a5c86e7c1de635a87beb8ed2ae6a2eb6886736e8d8f11179152" exitCode=0 Nov 01 23:13:05.161208 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:05.161038 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" event={"ID":"a29727f6-5841-404b-aa67-d596290c08d1","Type":"ContainerDied","Data":"061199c7094d1a5c86e7c1de635a87beb8ed2ae6a2eb6886736e8d8f11179152"} Nov 01 23:13:05.161208 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:05.161066 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" Nov 01 23:13:05.161208 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:05.161091 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml" event={"ID":"a29727f6-5841-404b-aa67-d596290c08d1","Type":"ContainerDied","Data":"ab5fd33d5d8913dd500c6cc094b245703314c486e09cf442cff1bf22f3071870"} Nov 01 23:13:05.161208 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:05.161112 5011 scope.go:122] "RemoveContainer" containerID="061199c7094d1a5c86e7c1de635a87beb8ed2ae6a2eb6886736e8d8f11179152" Nov 01 23:13:05.178356 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:05.176016 5011 scope.go:122] "RemoveContainer" containerID="061199c7094d1a5c86e7c1de635a87beb8ed2ae6a2eb6886736e8d8f11179152" Nov 01 23:13:05.178356 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:05.177562 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-66565fcd54-x4c6t" podStartSLOduration=1.554694934 podStartE2EDuration="2.177549701s" totalImagesPullingTime="622.854767ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:13:05.176273912 +0000 UTC m=+2953.580864545" watchObservedRunningTime="2026-11-01 23:13:05.177549701 +0000 UTC m=+2953.582140310" Nov 01 23:13:05.182928 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:13:05.182901 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"061199c7094d1a5c86e7c1de635a87beb8ed2ae6a2eb6886736e8d8f11179152\": container with ID starting with 061199c7094d1a5c86e7c1de635a87beb8ed2ae6a2eb6886736e8d8f11179152 not found: ID does not exist" containerID="061199c7094d1a5c86e7c1de635a87beb8ed2ae6a2eb6886736e8d8f11179152" Nov 01 23:13:05.183131 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:05.183102 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"061199c7094d1a5c86e7c1de635a87beb8ed2ae6a2eb6886736e8d8f11179152"} err="failed to get container status \"061199c7094d1a5c86e7c1de635a87beb8ed2ae6a2eb6886736e8d8f11179152\": rpc error: code = NotFound desc = could not find container \"061199c7094d1a5c86e7c1de635a87beb8ed2ae6a2eb6886736e8d8f11179152\": container with ID starting with 061199c7094d1a5c86e7c1de635a87beb8ed2ae6a2eb6886736e8d8f11179152 not found: ID does not exist" Nov 01 23:13:05.197624 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:05.197590 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml"] Nov 01 23:13:05.202370 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:05.201924 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-rhlml"] Nov 01 23:13:05.745948 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:05.745916 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a29727f6-5841-404b-aa67-d596290c08d1" path="/var/lib/kubelet/pods/a29727f6-5841-404b-aa67-d596290c08d1/volumes" Nov 01 23:13:16.169741 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:16.169712 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-66565fcd54-x4c6t" Nov 01 23:13:16.959620 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:13:16.959579 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:13:20.472308 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.472276 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l"] Nov 01 23:13:20.472940 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.472800 5011 cpu_manager.go:414] "containerMap: removing container" podUID="b0d043d9-49a8-42fb-872c-cd5479590b1f" containerName="istio-proxy" Nov 01 23:13:20.472940 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.472820 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0d043d9-49a8-42fb-872c-cd5479590b1f" containerName="istio-proxy" Nov 01 23:13:20.472940 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.472893 5011 cpu_manager.go:414] "containerMap: removing container" podUID="a29727f6-5841-404b-aa67-d596290c08d1" containerName="istio-proxy" Nov 01 23:13:20.472940 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.472902 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29727f6-5841-404b-aa67-d596290c08d1" containerName="istio-proxy" Nov 01 23:13:20.473192 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.473019 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0d043d9-49a8-42fb-872c-cd5479590b1f" containerName="istio-proxy" Nov 01 23:13:20.473192 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.473035 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29727f6-5841-404b-aa67-d596290c08d1" containerName="istio-proxy" Nov 01 23:13:20.476235 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.476215 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:20.489110 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.489082 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l"] Nov 01 23:13:20.652321 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.652293 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79p9r\" (UniqueName: \"kubernetes.io/projected/4688829c-8b38-4d0f-a56c-f20892644b09-kube-api-access-79p9r\") pod \"maas-default-gateway-openshift-default-6886b96f9-bxw6l\" (UID: \"4688829c-8b38-4d0f-a56c-f20892644b09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:20.652321 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.652327 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4688829c-8b38-4d0f-a56c-f20892644b09-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-bxw6l\" (UID: \"4688829c-8b38-4d0f-a56c-f20892644b09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:20.652549 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.652346 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4688829c-8b38-4d0f-a56c-f20892644b09-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-bxw6l\" (UID: \"4688829c-8b38-4d0f-a56c-f20892644b09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:20.652549 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.652486 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4688829c-8b38-4d0f-a56c-f20892644b09-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-bxw6l\" (UID: \"4688829c-8b38-4d0f-a56c-f20892644b09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:20.652549 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.652519 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4688829c-8b38-4d0f-a56c-f20892644b09-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-bxw6l\" (UID: \"4688829c-8b38-4d0f-a56c-f20892644b09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:20.652663 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.652558 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4688829c-8b38-4d0f-a56c-f20892644b09-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-bxw6l\" (UID: \"4688829c-8b38-4d0f-a56c-f20892644b09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:20.652663 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.652611 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4688829c-8b38-4d0f-a56c-f20892644b09-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-bxw6l\" (UID: \"4688829c-8b38-4d0f-a56c-f20892644b09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:20.652736 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.652728 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4688829c-8b38-4d0f-a56c-f20892644b09-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-bxw6l\" (UID: \"4688829c-8b38-4d0f-a56c-f20892644b09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:20.652771 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.652755 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4688829c-8b38-4d0f-a56c-f20892644b09-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-bxw6l\" (UID: \"4688829c-8b38-4d0f-a56c-f20892644b09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:20.753755 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.753662 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4688829c-8b38-4d0f-a56c-f20892644b09-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-bxw6l\" (UID: \"4688829c-8b38-4d0f-a56c-f20892644b09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:20.753755 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.753706 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4688829c-8b38-4d0f-a56c-f20892644b09-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-bxw6l\" (UID: \"4688829c-8b38-4d0f-a56c-f20892644b09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:20.753755 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.753725 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79p9r\" (UniqueName: \"kubernetes.io/projected/4688829c-8b38-4d0f-a56c-f20892644b09-kube-api-access-79p9r\") pod \"maas-default-gateway-openshift-default-6886b96f9-bxw6l\" (UID: \"4688829c-8b38-4d0f-a56c-f20892644b09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:20.753755 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.753744 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4688829c-8b38-4d0f-a56c-f20892644b09-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-bxw6l\" (UID: \"4688829c-8b38-4d0f-a56c-f20892644b09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:20.754157 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.753770 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4688829c-8b38-4d0f-a56c-f20892644b09-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-bxw6l\" (UID: \"4688829c-8b38-4d0f-a56c-f20892644b09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:20.754157 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.753806 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4688829c-8b38-4d0f-a56c-f20892644b09-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-bxw6l\" (UID: \"4688829c-8b38-4d0f-a56c-f20892644b09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:20.754157 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.753868 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4688829c-8b38-4d0f-a56c-f20892644b09-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-bxw6l\" (UID: \"4688829c-8b38-4d0f-a56c-f20892644b09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:20.754157 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.753906 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4688829c-8b38-4d0f-a56c-f20892644b09-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-bxw6l\" (UID: \"4688829c-8b38-4d0f-a56c-f20892644b09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:20.754157 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.753937 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4688829c-8b38-4d0f-a56c-f20892644b09-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-bxw6l\" (UID: \"4688829c-8b38-4d0f-a56c-f20892644b09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:20.754408 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.754208 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4688829c-8b38-4d0f-a56c-f20892644b09-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-bxw6l\" (UID: \"4688829c-8b38-4d0f-a56c-f20892644b09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:20.754408 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.754234 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4688829c-8b38-4d0f-a56c-f20892644b09-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-bxw6l\" (UID: \"4688829c-8b38-4d0f-a56c-f20892644b09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:20.754408 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.754350 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4688829c-8b38-4d0f-a56c-f20892644b09-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-bxw6l\" (UID: \"4688829c-8b38-4d0f-a56c-f20892644b09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:20.754588 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.754485 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4688829c-8b38-4d0f-a56c-f20892644b09-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-bxw6l\" (UID: \"4688829c-8b38-4d0f-a56c-f20892644b09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:20.754761 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.754742 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4688829c-8b38-4d0f-a56c-f20892644b09-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-bxw6l\" (UID: \"4688829c-8b38-4d0f-a56c-f20892644b09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:20.756622 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.756599 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4688829c-8b38-4d0f-a56c-f20892644b09-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-bxw6l\" (UID: \"4688829c-8b38-4d0f-a56c-f20892644b09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:20.756622 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.756611 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4688829c-8b38-4d0f-a56c-f20892644b09-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-bxw6l\" (UID: \"4688829c-8b38-4d0f-a56c-f20892644b09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:20.762453 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.762422 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4688829c-8b38-4d0f-a56c-f20892644b09-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-bxw6l\" (UID: \"4688829c-8b38-4d0f-a56c-f20892644b09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:20.762589 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.762571 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79p9r\" (UniqueName: \"kubernetes.io/projected/4688829c-8b38-4d0f-a56c-f20892644b09-kube-api-access-79p9r\") pod \"maas-default-gateway-openshift-default-6886b96f9-bxw6l\" (UID: \"4688829c-8b38-4d0f-a56c-f20892644b09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:20.789983 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:20.789935 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:21.070108 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:21.069983 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l"] Nov 01 23:13:21.073106 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:13:21.073076 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4688829c_8b38_4d0f_a56c_f20892644b09.slice/crio-0c02bca6417d163dfe55a63c810228ee7307fe8fd92d715977cc2cc85440e1d5 WatchSource:0}: Error finding container 0c02bca6417d163dfe55a63c810228ee7307fe8fd92d715977cc2cc85440e1d5: Status 404 returned error can't find the container with id 0c02bca6417d163dfe55a63c810228ee7307fe8fd92d715977cc2cc85440e1d5 Nov 01 23:13:21.075423 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:21.075387 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:13:21.075522 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:21.075459 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:13:21.075522 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:21.075489 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:13:21.247144 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:21.247103 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" event={"ID":"4688829c-8b38-4d0f-a56c-f20892644b09","Type":"ContainerStarted","Data":"02f837ab30cba8faf234ebb12c82ff44cd12f13e7f2d9e69e0bba89a8e9b195b"} Nov 01 23:13:21.247362 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:21.247173 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" event={"ID":"4688829c-8b38-4d0f-a56c-f20892644b09","Type":"ContainerStarted","Data":"0c02bca6417d163dfe55a63c810228ee7307fe8fd92d715977cc2cc85440e1d5"} Nov 01 23:13:21.269276 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:21.269214 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" podStartSLOduration=1.269193182 podStartE2EDuration="1.269193182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:13:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:13:21.268105139 +0000 UTC m=+2969.672695772" watchObservedRunningTime="2026-11-01 23:13:21.269193182 +0000 UTC m=+2969.673783815" Nov 01 23:13:21.481109 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:21.481065 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gk9kf"] Nov 01 23:13:21.498179 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:21.498144 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gk9kf"] Nov 01 23:13:21.498408 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:21.498291 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gk9kf" Nov 01 23:13:21.502398 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:21.502368 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:13:21.502618 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:21.502373 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:13:21.502794 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:21.502621 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:13:21.502794 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:21.502669 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:13:21.671068 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:21.671036 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbsw2\" (UniqueName: \"kubernetes.io/projected/a6078069-ba77-49f3-bf44-58db4e58e202-kube-api-access-nbsw2\") pod \"maas-api-revoke-sub-85237e45-gk9kf\" (UID: \"a6078069-ba77-49f3-bf44-58db4e58e202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gk9kf" Nov 01 23:13:21.671068 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:21.671077 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6078069-ba77-49f3-bf44-58db4e58e202-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-gk9kf\" (UID: \"a6078069-ba77-49f3-bf44-58db4e58e202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gk9kf" Nov 01 23:13:21.671349 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:21.671284 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6078069-ba77-49f3-bf44-58db4e58e202-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-gk9kf\" (UID: \"a6078069-ba77-49f3-bf44-58db4e58e202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gk9kf" Nov 01 23:13:21.774297 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:21.774195 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbsw2\" (UniqueName: \"kubernetes.io/projected/a6078069-ba77-49f3-bf44-58db4e58e202-kube-api-access-nbsw2\") pod \"maas-api-revoke-sub-85237e45-gk9kf\" (UID: \"a6078069-ba77-49f3-bf44-58db4e58e202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gk9kf" Nov 01 23:13:21.774297 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:21.774255 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6078069-ba77-49f3-bf44-58db4e58e202-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-gk9kf\" (UID: \"a6078069-ba77-49f3-bf44-58db4e58e202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gk9kf" Nov 01 23:13:21.774556 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:21.774400 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6078069-ba77-49f3-bf44-58db4e58e202-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-gk9kf\" (UID: \"a6078069-ba77-49f3-bf44-58db4e58e202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gk9kf" Nov 01 23:13:21.775145 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:21.775116 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6078069-ba77-49f3-bf44-58db4e58e202-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-gk9kf\" (UID: \"a6078069-ba77-49f3-bf44-58db4e58e202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gk9kf" Nov 01 23:13:21.777311 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:21.777286 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6078069-ba77-49f3-bf44-58db4e58e202-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-gk9kf\" (UID: \"a6078069-ba77-49f3-bf44-58db4e58e202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gk9kf" Nov 01 23:13:21.781959 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:21.781923 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbsw2\" (UniqueName: \"kubernetes.io/projected/a6078069-ba77-49f3-bf44-58db4e58e202-kube-api-access-nbsw2\") pod \"maas-api-revoke-sub-85237e45-gk9kf\" (UID: \"a6078069-ba77-49f3-bf44-58db4e58e202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gk9kf" Nov 01 23:13:21.790869 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:21.790842 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:21.815046 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:21.815022 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gk9kf" Nov 01 23:13:22.119782 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:22.119754 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gk9kf"] Nov 01 23:13:22.122085 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:13:22.122060 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6078069_ba77_49f3_bf44_58db4e58e202.slice/crio-4044d81e823c6023363d269a745e279f69ab9e9644b23772035a60db84091c56 WatchSource:0}: Error finding container 4044d81e823c6023363d269a745e279f69ab9e9644b23772035a60db84091c56: Status 404 returned error can't find the container with id 4044d81e823c6023363d269a745e279f69ab9e9644b23772035a60db84091c56 Nov 01 23:13:22.270983 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:22.270950 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gk9kf" event={"ID":"a6078069-ba77-49f3-bf44-58db4e58e202","Type":"ContainerStarted","Data":"4044d81e823c6023363d269a745e279f69ab9e9644b23772035a60db84091c56"} Nov 01 23:13:22.792376 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:22.792348 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bxw6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 01 23:13:22.792736 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:22.792395 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" podUID="4688829c-8b38-4d0f-a56c-f20892644b09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 01 23:13:23.280412 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:23.280377 5011 generic.go:356] "Generic (PLEG): container finished" podID="a6078069-ba77-49f3-bf44-58db4e58e202" containerID="0b941e0282da1cc866b88d6de8ab71f2322fbcb83cd5a3768809a8c081d18458" exitCode=0 Nov 01 23:13:23.280601 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:23.280450 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gk9kf" event={"ID":"a6078069-ba77-49f3-bf44-58db4e58e202","Type":"ContainerDied","Data":"0b941e0282da1cc866b88d6de8ab71f2322fbcb83cd5a3768809a8c081d18458"} Nov 01 23:13:23.791264 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:23.791229 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bxw6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 01 23:13:23.791449 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:23.791281 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" podUID="4688829c-8b38-4d0f-a56c-f20892644b09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 01 23:13:24.431645 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:24.431619 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gk9kf" Nov 01 23:13:24.603094 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:24.603014 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6078069-ba77-49f3-bf44-58db4e58e202-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6078069-ba77-49f3-bf44-58db4e58e202-maas-api-xks-ca\") pod \"a6078069-ba77-49f3-bf44-58db4e58e202\" (UID: \"a6078069-ba77-49f3-bf44-58db4e58e202\") " Nov 01 23:13:24.603257 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:24.603105 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6078069-ba77-49f3-bf44-58db4e58e202-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6078069-ba77-49f3-bf44-58db4e58e202-maas-api-openshift-ca\") pod \"a6078069-ba77-49f3-bf44-58db4e58e202\" (UID: \"a6078069-ba77-49f3-bf44-58db4e58e202\") " Nov 01 23:13:24.603310 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:24.603274 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6078069-ba77-49f3-bf44-58db4e58e202-kube-api-access-nbsw2\" (UniqueName: \"kubernetes.io/projected/a6078069-ba77-49f3-bf44-58db4e58e202-kube-api-access-nbsw2\") pod \"a6078069-ba77-49f3-bf44-58db4e58e202\" (UID: \"a6078069-ba77-49f3-bf44-58db4e58e202\") " Nov 01 23:13:24.603410 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:24.603390 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6078069-ba77-49f3-bf44-58db4e58e202-maas-api-openshift-ca" pod "a6078069-ba77-49f3-bf44-58db4e58e202" (UID: "a6078069-ba77-49f3-bf44-58db4e58e202"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:13:24.603602 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:24.603586 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6078069-ba77-49f3-bf44-58db4e58e202-maas-api-openshift-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:13:24.605433 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:24.605407 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6078069-ba77-49f3-bf44-58db4e58e202-maas-api-xks-ca" pod "a6078069-ba77-49f3-bf44-58db4e58e202" (UID: "a6078069-ba77-49f3-bf44-58db4e58e202"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:13:24.605558 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:24.605507 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6078069-ba77-49f3-bf44-58db4e58e202-kube-api-access-nbsw2" pod "a6078069-ba77-49f3-bf44-58db4e58e202" (UID: "a6078069-ba77-49f3-bf44-58db4e58e202"). InnerVolumeSpecName "kube-api-access-nbsw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:13:24.704489 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:24.704456 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6078069-ba77-49f3-bf44-58db4e58e202-maas-api-xks-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:13:24.704489 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:24.704488 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbsw2\" (UniqueName: \"kubernetes.io/projected/a6078069-ba77-49f3-bf44-58db4e58e202-kube-api-access-nbsw2\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:13:24.791371 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:24.791327 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bxw6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 01 23:13:24.791568 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:24.791396 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" podUID="4688829c-8b38-4d0f-a56c-f20892644b09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 01 23:13:25.291100 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:25.291066 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gk9kf" event={"ID":"a6078069-ba77-49f3-bf44-58db4e58e202","Type":"ContainerDied","Data":"4044d81e823c6023363d269a745e279f69ab9e9644b23772035a60db84091c56"} Nov 01 23:13:25.291297 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:25.291102 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gk9kf" Nov 01 23:13:25.291297 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:25.291115 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4044d81e823c6023363d269a745e279f69ab9e9644b23772035a60db84091c56" Nov 01 23:13:25.791291 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:25.791255 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bxw6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 01 23:13:25.791721 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:25.791309 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" podUID="4688829c-8b38-4d0f-a56c-f20892644b09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 01 23:13:26.790324 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:26.790290 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bxw6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 01 23:13:26.790528 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:26.790349 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" podUID="4688829c-8b38-4d0f-a56c-f20892644b09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 01 23:13:27.790683 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:27.790641 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bxw6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Nov 01 23:13:27.791264 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:27.790696 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" podUID="4688829c-8b38-4d0f-a56c-f20892644b09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Nov 01 23:13:29.790878 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:29.790809 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bxw6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:13:29.791424 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:29.790895 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" podUID="4688829c-8b38-4d0f-a56c-f20892644b09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:13:30.832078 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:30.832030 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bxw6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:13:30.832536 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:30.832103 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" podUID="4688829c-8b38-4d0f-a56c-f20892644b09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:13:31.379381 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:31.379351 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:31.379674 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:31.379651 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:31.380671 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:31.380650 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" Nov 01 23:13:33.034684 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:33.034650 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66565fcd54-x4c6t"] Nov 01 23:13:33.035241 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:33.034962 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-66565fcd54-x4c6t" podUID="f4631edc-4126-4c12-85a6-8286a976d741" containerName="manager" containerID="cri-o://85414a9e94843a9e8eab7e5b74b620dde38ab7be0200e687df4a9307b6ee271e" gracePeriod=30 Nov 01 23:13:33.279009 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:33.278979 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66565fcd54-x4c6t" Nov 01 23:13:33.331330 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:33.331226 5011 generic.go:356] "Generic (PLEG): container finished" podID="f4631edc-4126-4c12-85a6-8286a976d741" containerID="85414a9e94843a9e8eab7e5b74b620dde38ab7be0200e687df4a9307b6ee271e" exitCode=0 Nov 01 23:13:33.331330 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:33.331277 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66565fcd54-x4c6t" Nov 01 23:13:33.331595 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:33.331321 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66565fcd54-x4c6t" event={"ID":"f4631edc-4126-4c12-85a6-8286a976d741","Type":"ContainerDied","Data":"85414a9e94843a9e8eab7e5b74b620dde38ab7be0200e687df4a9307b6ee271e"} Nov 01 23:13:33.331595 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:33.331387 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66565fcd54-x4c6t" event={"ID":"f4631edc-4126-4c12-85a6-8286a976d741","Type":"ContainerDied","Data":"fed856f6871116c67d0371b5196d3bed1383dc967957f6f341866ce788338c0e"} Nov 01 23:13:33.331595 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:33.331411 5011 scope.go:122] "RemoveContainer" containerID="85414a9e94843a9e8eab7e5b74b620dde38ab7be0200e687df4a9307b6ee271e" Nov 01 23:13:33.342367 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:33.342346 5011 scope.go:122] "RemoveContainer" containerID="85414a9e94843a9e8eab7e5b74b620dde38ab7be0200e687df4a9307b6ee271e" Nov 01 23:13:33.342713 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:13:33.342690 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85414a9e94843a9e8eab7e5b74b620dde38ab7be0200e687df4a9307b6ee271e\": container with ID starting with 85414a9e94843a9e8eab7e5b74b620dde38ab7be0200e687df4a9307b6ee271e not found: ID does not exist" containerID="85414a9e94843a9e8eab7e5b74b620dde38ab7be0200e687df4a9307b6ee271e" Nov 01 23:13:33.342807 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:33.342722 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85414a9e94843a9e8eab7e5b74b620dde38ab7be0200e687df4a9307b6ee271e"} err="failed to get container status \"85414a9e94843a9e8eab7e5b74b620dde38ab7be0200e687df4a9307b6ee271e\": rpc error: code = NotFound desc = could not find container \"85414a9e94843a9e8eab7e5b74b620dde38ab7be0200e687df4a9307b6ee271e\": container with ID starting with 85414a9e94843a9e8eab7e5b74b620dde38ab7be0200e687df4a9307b6ee271e not found: ID does not exist" Nov 01 23:13:33.380034 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:33.380000 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4631edc-4126-4c12-85a6-8286a976d741-kube-api-access-7cjtz\" (UniqueName: \"kubernetes.io/projected/f4631edc-4126-4c12-85a6-8286a976d741-kube-api-access-7cjtz\") pod \"f4631edc-4126-4c12-85a6-8286a976d741\" (UID: \"f4631edc-4126-4c12-85a6-8286a976d741\") " Nov 01 23:13:33.380214 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:33.380069 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4631edc-4126-4c12-85a6-8286a976d741-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4631edc-4126-4c12-85a6-8286a976d741-metrics-certs\") pod \"f4631edc-4126-4c12-85a6-8286a976d741\" (UID: \"f4631edc-4126-4c12-85a6-8286a976d741\") " Nov 01 23:13:33.380214 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:33.380137 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4631edc-4126-4c12-85a6-8286a976d741-webhook-cert\" (UniqueName: \"kubernetes.io/secret/f4631edc-4126-4c12-85a6-8286a976d741-webhook-cert\") pod \"f4631edc-4126-4c12-85a6-8286a976d741\" (UID: \"f4631edc-4126-4c12-85a6-8286a976d741\") " Nov 01 23:13:33.382491 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:33.382438 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4631edc-4126-4c12-85a6-8286a976d741-metrics-certs" pod "f4631edc-4126-4c12-85a6-8286a976d741" (UID: "f4631edc-4126-4c12-85a6-8286a976d741"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:13:33.382491 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:33.382459 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4631edc-4126-4c12-85a6-8286a976d741-webhook-cert" pod "f4631edc-4126-4c12-85a6-8286a976d741" (UID: "f4631edc-4126-4c12-85a6-8286a976d741"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:13:33.382491 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:33.382489 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4631edc-4126-4c12-85a6-8286a976d741-kube-api-access-7cjtz" pod "f4631edc-4126-4c12-85a6-8286a976d741" (UID: "f4631edc-4126-4c12-85a6-8286a976d741"). InnerVolumeSpecName "kube-api-access-7cjtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:13:33.481333 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:33.481298 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cjtz\" (UniqueName: \"kubernetes.io/projected/f4631edc-4126-4c12-85a6-8286a976d741-kube-api-access-7cjtz\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:13:33.481333 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:33.481330 5011 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4631edc-4126-4c12-85a6-8286a976d741-metrics-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:13:33.481333 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:33.481339 5011 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f4631edc-4126-4c12-85a6-8286a976d741-webhook-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:13:33.652701 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:33.652665 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66565fcd54-x4c6t"] Nov 01 23:13:33.654573 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:33.654550 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-66565fcd54-x4c6t"] Nov 01 23:13:33.744860 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:33.744799 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4631edc-4126-4c12-85a6-8286a976d741" path="/var/lib/kubelet/pods/f4631edc-4126-4c12-85a6-8286a976d741/volumes" Nov 01 23:13:34.820018 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:34.819983 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-66565fcd54-4b495"] Nov 01 23:13:34.820425 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:34.820336 5011 cpu_manager.go:414] "containerMap: removing container" podUID="a6078069-ba77-49f3-bf44-58db4e58e202" containerName="revoke-keys" Nov 01 23:13:34.820425 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:34.820347 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6078069-ba77-49f3-bf44-58db4e58e202" containerName="revoke-keys" Nov 01 23:13:34.820425 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:34.820368 5011 cpu_manager.go:414] "containerMap: removing container" podUID="f4631edc-4126-4c12-85a6-8286a976d741" containerName="manager" Nov 01 23:13:34.820425 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:34.820374 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4631edc-4126-4c12-85a6-8286a976d741" containerName="manager" Nov 01 23:13:34.820560 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:34.820433 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4631edc-4126-4c12-85a6-8286a976d741" containerName="manager" Nov 01 23:13:34.820560 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:34.820445 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6078069-ba77-49f3-bf44-58db4e58e202" containerName="revoke-keys" Nov 01 23:13:34.825102 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:34.825080 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66565fcd54-4b495" Nov 01 23:13:34.833152 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:34.833116 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 01 23:13:34.833321 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:34.833174 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 01 23:13:34.833321 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:34.833230 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ztmdx\"" Nov 01 23:13:34.833321 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:34.833253 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 01 23:13:34.835338 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:34.835319 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66565fcd54-4b495"] Nov 01 23:13:34.893754 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:34.893723 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c0b5bb7-8733-417a-9981-475611b520df-metrics-certs\") pod \"maas-controller-66565fcd54-4b495\" (UID: \"1c0b5bb7-8733-417a-9981-475611b520df\") " pod="opendatahub/maas-controller-66565fcd54-4b495" Nov 01 23:13:34.893754 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:34.893764 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4l58\" (UniqueName: \"kubernetes.io/projected/1c0b5bb7-8733-417a-9981-475611b520df-kube-api-access-r4l58\") pod \"maas-controller-66565fcd54-4b495\" (UID: \"1c0b5bb7-8733-417a-9981-475611b520df\") " pod="opendatahub/maas-controller-66565fcd54-4b495" Nov 01 23:13:34.894070 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:34.893941 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c0b5bb7-8733-417a-9981-475611b520df-webhook-cert\") pod \"maas-controller-66565fcd54-4b495\" (UID: \"1c0b5bb7-8733-417a-9981-475611b520df\") " pod="opendatahub/maas-controller-66565fcd54-4b495" Nov 01 23:13:34.995019 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:34.994966 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c0b5bb7-8733-417a-9981-475611b520df-webhook-cert\") pod \"maas-controller-66565fcd54-4b495\" (UID: \"1c0b5bb7-8733-417a-9981-475611b520df\") " pod="opendatahub/maas-controller-66565fcd54-4b495" Nov 01 23:13:34.995211 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:34.995062 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c0b5bb7-8733-417a-9981-475611b520df-metrics-certs\") pod \"maas-controller-66565fcd54-4b495\" (UID: \"1c0b5bb7-8733-417a-9981-475611b520df\") " pod="opendatahub/maas-controller-66565fcd54-4b495" Nov 01 23:13:34.995211 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:34.995082 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4l58\" (UniqueName: \"kubernetes.io/projected/1c0b5bb7-8733-417a-9981-475611b520df-kube-api-access-r4l58\") pod \"maas-controller-66565fcd54-4b495\" (UID: \"1c0b5bb7-8733-417a-9981-475611b520df\") " pod="opendatahub/maas-controller-66565fcd54-4b495" Nov 01 23:13:34.997666 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:34.997635 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c0b5bb7-8733-417a-9981-475611b520df-metrics-certs\") pod \"maas-controller-66565fcd54-4b495\" (UID: \"1c0b5bb7-8733-417a-9981-475611b520df\") " pod="opendatahub/maas-controller-66565fcd54-4b495" Nov 01 23:13:34.997782 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:34.997695 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c0b5bb7-8733-417a-9981-475611b520df-webhook-cert\") pod \"maas-controller-66565fcd54-4b495\" (UID: \"1c0b5bb7-8733-417a-9981-475611b520df\") " pod="opendatahub/maas-controller-66565fcd54-4b495" Nov 01 23:13:35.003424 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.003400 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4l58\" (UniqueName: \"kubernetes.io/projected/1c0b5bb7-8733-417a-9981-475611b520df-kube-api-access-r4l58\") pod \"maas-controller-66565fcd54-4b495\" (UID: \"1c0b5bb7-8733-417a-9981-475611b520df\") " pod="opendatahub/maas-controller-66565fcd54-4b495" Nov 01 23:13:35.139519 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.139438 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66565fcd54-4b495" Nov 01 23:13:35.412749 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.412724 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66565fcd54-4b495"] Nov 01 23:13:35.415390 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:13:35.415363 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c0b5bb7_8733_417a_9981_475611b520df.slice/crio-c18f7f9ef0fe00e7a1992c9bba665f84633469111d2f55f9a92906546b0cad74 WatchSource:0}: Error finding container c18f7f9ef0fe00e7a1992c9bba665f84633469111d2f55f9a92906546b0cad74: Status 404 returned error can't find the container with id c18f7f9ef0fe00e7a1992c9bba665f84633469111d2f55f9a92906546b0cad74 Nov 01 23:13:35.487786 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.487752 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr"] Nov 01 23:13:35.492652 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.492628 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:35.502542 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.502507 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr"] Nov 01 23:13:35.600804 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.600769 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9832d7-c187-41b2-bf9d-de25b0ea455a-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-m4nvr\" (UID: \"fd9832d7-c187-41b2-bf9d-de25b0ea455a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:35.601002 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.600817 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9832d7-c187-41b2-bf9d-de25b0ea455a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-m4nvr\" (UID: \"fd9832d7-c187-41b2-bf9d-de25b0ea455a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:35.601002 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.600865 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9832d7-c187-41b2-bf9d-de25b0ea455a-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-m4nvr\" (UID: \"fd9832d7-c187-41b2-bf9d-de25b0ea455a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:35.601002 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.600912 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9832d7-c187-41b2-bf9d-de25b0ea455a-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-m4nvr\" (UID: \"fd9832d7-c187-41b2-bf9d-de25b0ea455a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:35.601128 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.601014 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9832d7-c187-41b2-bf9d-de25b0ea455a-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-m4nvr\" (UID: \"fd9832d7-c187-41b2-bf9d-de25b0ea455a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:35.601128 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.601039 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9832d7-c187-41b2-bf9d-de25b0ea455a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-m4nvr\" (UID: \"fd9832d7-c187-41b2-bf9d-de25b0ea455a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:35.601128 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.601070 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9832d7-c187-41b2-bf9d-de25b0ea455a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-m4nvr\" (UID: \"fd9832d7-c187-41b2-bf9d-de25b0ea455a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:35.601239 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.601137 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zgbf\" (UniqueName: \"kubernetes.io/projected/fd9832d7-c187-41b2-bf9d-de25b0ea455a-kube-api-access-5zgbf\") pod \"maas-default-gateway-openshift-default-6886b96f9-m4nvr\" (UID: \"fd9832d7-c187-41b2-bf9d-de25b0ea455a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:35.601239 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.601170 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9832d7-c187-41b2-bf9d-de25b0ea455a-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-m4nvr\" (UID: \"fd9832d7-c187-41b2-bf9d-de25b0ea455a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:35.702496 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.702458 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9832d7-c187-41b2-bf9d-de25b0ea455a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-m4nvr\" (UID: \"fd9832d7-c187-41b2-bf9d-de25b0ea455a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:35.702671 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.702505 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zgbf\" (UniqueName: \"kubernetes.io/projected/fd9832d7-c187-41b2-bf9d-de25b0ea455a-kube-api-access-5zgbf\") pod \"maas-default-gateway-openshift-default-6886b96f9-m4nvr\" (UID: \"fd9832d7-c187-41b2-bf9d-de25b0ea455a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:35.702671 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.702548 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9832d7-c187-41b2-bf9d-de25b0ea455a-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-m4nvr\" (UID: \"fd9832d7-c187-41b2-bf9d-de25b0ea455a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:35.702671 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.702578 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9832d7-c187-41b2-bf9d-de25b0ea455a-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-m4nvr\" (UID: \"fd9832d7-c187-41b2-bf9d-de25b0ea455a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:35.702671 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.702612 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9832d7-c187-41b2-bf9d-de25b0ea455a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-m4nvr\" (UID: \"fd9832d7-c187-41b2-bf9d-de25b0ea455a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:35.702671 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.702653 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9832d7-c187-41b2-bf9d-de25b0ea455a-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-m4nvr\" (UID: \"fd9832d7-c187-41b2-bf9d-de25b0ea455a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:35.703030 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.702676 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9832d7-c187-41b2-bf9d-de25b0ea455a-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-m4nvr\" (UID: \"fd9832d7-c187-41b2-bf9d-de25b0ea455a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:35.703030 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.702731 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9832d7-c187-41b2-bf9d-de25b0ea455a-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-m4nvr\" (UID: \"fd9832d7-c187-41b2-bf9d-de25b0ea455a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:35.703030 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.702758 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9832d7-c187-41b2-bf9d-de25b0ea455a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-m4nvr\" (UID: \"fd9832d7-c187-41b2-bf9d-de25b0ea455a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:35.703204 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.703147 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9832d7-c187-41b2-bf9d-de25b0ea455a-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-m4nvr\" (UID: \"fd9832d7-c187-41b2-bf9d-de25b0ea455a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:35.703204 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.703187 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9832d7-c187-41b2-bf9d-de25b0ea455a-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-m4nvr\" (UID: \"fd9832d7-c187-41b2-bf9d-de25b0ea455a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:35.703382 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.703256 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9832d7-c187-41b2-bf9d-de25b0ea455a-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-m4nvr\" (UID: \"fd9832d7-c187-41b2-bf9d-de25b0ea455a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:35.703382 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.703271 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9832d7-c187-41b2-bf9d-de25b0ea455a-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-m4nvr\" (UID: \"fd9832d7-c187-41b2-bf9d-de25b0ea455a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:35.703495 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.703474 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9832d7-c187-41b2-bf9d-de25b0ea455a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-m4nvr\" (UID: \"fd9832d7-c187-41b2-bf9d-de25b0ea455a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:35.705223 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.705205 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9832d7-c187-41b2-bf9d-de25b0ea455a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-m4nvr\" (UID: \"fd9832d7-c187-41b2-bf9d-de25b0ea455a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:35.705395 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.705376 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9832d7-c187-41b2-bf9d-de25b0ea455a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-m4nvr\" (UID: \"fd9832d7-c187-41b2-bf9d-de25b0ea455a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:35.711192 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.711169 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9832d7-c187-41b2-bf9d-de25b0ea455a-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-m4nvr\" (UID: \"fd9832d7-c187-41b2-bf9d-de25b0ea455a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:35.711348 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.711330 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zgbf\" (UniqueName: \"kubernetes.io/projected/fd9832d7-c187-41b2-bf9d-de25b0ea455a-kube-api-access-5zgbf\") pod \"maas-default-gateway-openshift-default-6886b96f9-m4nvr\" (UID: \"fd9832d7-c187-41b2-bf9d-de25b0ea455a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:35.808150 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:35.808112 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:36.085737 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:36.085708 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr"] Nov 01 23:13:36.087399 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:13:36.087377 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd9832d7_c187_41b2_bf9d_de25b0ea455a.slice/crio-ba9c64b249e6abb6608a10e7b71b0c4b326121ab131d7dcb18695f6957c205fd WatchSource:0}: Error finding container ba9c64b249e6abb6608a10e7b71b0c4b326121ab131d7dcb18695f6957c205fd: Status 404 returned error can't find the container with id ba9c64b249e6abb6608a10e7b71b0c4b326121ab131d7dcb18695f6957c205fd Nov 01 23:13:36.090162 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:36.090118 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:13:36.090248 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:36.090221 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:13:36.090292 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:36.090265 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:13:36.349899 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:36.349777 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" event={"ID":"fd9832d7-c187-41b2-bf9d-de25b0ea455a","Type":"ContainerStarted","Data":"f284d854cc426892f20793986b701e5559d4a30705f1d340c3ab71c07228d77d"} Nov 01 23:13:36.349899 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:36.349868 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" event={"ID":"fd9832d7-c187-41b2-bf9d-de25b0ea455a","Type":"ContainerStarted","Data":"ba9c64b249e6abb6608a10e7b71b0c4b326121ab131d7dcb18695f6957c205fd"} Nov 01 23:13:36.351823 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:36.351791 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66565fcd54-4b495" event={"ID":"1c0b5bb7-8733-417a-9981-475611b520df","Type":"ContainerStarted","Data":"0c112229b57ed5ecb0d805e1b42af2ad3d95559dbdffb7b51fdb3d18d2e6bb51"} Nov 01 23:13:36.351988 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:36.351862 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66565fcd54-4b495" event={"ID":"1c0b5bb7-8733-417a-9981-475611b520df","Type":"ContainerStarted","Data":"c18f7f9ef0fe00e7a1992c9bba665f84633469111d2f55f9a92906546b0cad74"} Nov 01 23:13:36.351988 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:36.351972 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-66565fcd54-4b495" Nov 01 23:13:36.373755 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:36.373693 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" podStartSLOduration=1.373673085 podStartE2EDuration="1.373673085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:13:36.371203399 +0000 UTC m=+2984.775794055" watchObservedRunningTime="2026-11-01 23:13:36.373673085 +0000 UTC m=+2984.778263718" Nov 01 23:13:36.391252 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:36.391193 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-66565fcd54-4b495" podStartSLOduration=1.783752206 podStartE2EDuration="2.391179435s" totalImagesPullingTime="607.427229ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:13:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:13:36.389752631 +0000 UTC m=+2984.794343264" watchObservedRunningTime="2026-11-01 23:13:36.391179435 +0000 UTC m=+2984.795770081" Nov 01 23:13:36.809172 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:36.809137 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:37.810607 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:37.810568 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-m4nvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Nov 01 23:13:37.811121 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:37.810639 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" podUID="fd9832d7-c187-41b2-bf9d-de25b0ea455a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Nov 01 23:13:38.808632 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:38.808592 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-m4nvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Nov 01 23:13:38.808859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:38.808654 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" podUID="fd9832d7-c187-41b2-bf9d-de25b0ea455a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Nov 01 23:13:39.809448 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:39.809411 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-m4nvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Nov 01 23:13:39.809903 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:39.809464 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" podUID="fd9832d7-c187-41b2-bf9d-de25b0ea455a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Nov 01 23:13:40.809495 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:40.809457 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-m4nvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Nov 01 23:13:40.809945 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:40.809510 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" podUID="fd9832d7-c187-41b2-bf9d-de25b0ea455a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Nov 01 23:13:41.809465 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:41.809425 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-m4nvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Nov 01 23:13:41.809688 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:41.809503 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" podUID="fd9832d7-c187-41b2-bf9d-de25b0ea455a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Nov 01 23:13:42.809302 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:42.809262 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-m4nvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Nov 01 23:13:42.809492 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:42.809330 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" podUID="fd9832d7-c187-41b2-bf9d-de25b0ea455a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Nov 01 23:13:44.321972 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:44.321941 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:44.322418 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:44.322262 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:44.323108 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:44.323082 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" Nov 01 23:13:46.959712 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:13:46.959676 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:13:47.372893 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:13:47.372794 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-66565fcd54-4b495" Nov 01 23:14:09.011236 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:09.011196 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:14:13.507160 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:13.507121 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lsr22"] Nov 01 23:14:13.512158 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:13.512137 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lsr22" Nov 01 23:14:13.516054 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:13.516003 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:14:13.516999 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:13.516977 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:14:13.517140 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:13.517015 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:14:13.517140 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:13.517125 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:14:13.519444 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:13.519420 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lsr22"] Nov 01 23:14:13.641762 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:13.641724 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccca4473-3959-4e16-af06-0133607bc80d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-lsr22\" (UID: \"ccca4473-3959-4e16-af06-0133607bc80d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lsr22" Nov 01 23:14:13.641961 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:13.641774 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccca4473-3959-4e16-af06-0133607bc80d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-lsr22\" (UID: \"ccca4473-3959-4e16-af06-0133607bc80d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lsr22" Nov 01 23:14:13.642028 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:13.641951 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctswt\" (UniqueName: \"kubernetes.io/projected/ccca4473-3959-4e16-af06-0133607bc80d-kube-api-access-ctswt\") pod \"maas-api-revoke-sub-c1e271a9-lsr22\" (UID: \"ccca4473-3959-4e16-af06-0133607bc80d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lsr22" Nov 01 23:14:13.742422 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:13.742393 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctswt\" (UniqueName: \"kubernetes.io/projected/ccca4473-3959-4e16-af06-0133607bc80d-kube-api-access-ctswt\") pod \"maas-api-revoke-sub-c1e271a9-lsr22\" (UID: \"ccca4473-3959-4e16-af06-0133607bc80d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lsr22" Nov 01 23:14:13.742634 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:13.742449 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccca4473-3959-4e16-af06-0133607bc80d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-lsr22\" (UID: \"ccca4473-3959-4e16-af06-0133607bc80d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lsr22" Nov 01 23:14:13.742634 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:13.742474 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccca4473-3959-4e16-af06-0133607bc80d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-lsr22\" (UID: \"ccca4473-3959-4e16-af06-0133607bc80d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lsr22" Nov 01 23:14:13.743159 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:13.743133 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccca4473-3959-4e16-af06-0133607bc80d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-lsr22\" (UID: \"ccca4473-3959-4e16-af06-0133607bc80d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lsr22" Nov 01 23:14:13.745396 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:13.745375 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccca4473-3959-4e16-af06-0133607bc80d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-lsr22\" (UID: \"ccca4473-3959-4e16-af06-0133607bc80d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lsr22" Nov 01 23:14:13.750846 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:13.750812 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctswt\" (UniqueName: \"kubernetes.io/projected/ccca4473-3959-4e16-af06-0133607bc80d-kube-api-access-ctswt\") pod \"maas-api-revoke-sub-c1e271a9-lsr22\" (UID: \"ccca4473-3959-4e16-af06-0133607bc80d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lsr22" Nov 01 23:14:13.826163 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:13.826091 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lsr22" Nov 01 23:14:14.098889 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:14.098864 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lsr22"] Nov 01 23:14:14.101103 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:14:14.101069 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccca4473_3959_4e16_af06_0133607bc80d.slice/crio-08b5f32c69eda5ebe68efa207cd87659be76c7016081f5300f189fd89ae2ff54 WatchSource:0}: Error finding container 08b5f32c69eda5ebe68efa207cd87659be76c7016081f5300f189fd89ae2ff54: Status 404 returned error can't find the container with id 08b5f32c69eda5ebe68efa207cd87659be76c7016081f5300f189fd89ae2ff54 Nov 01 23:14:14.574222 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:14.574179 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lsr22" event={"ID":"ccca4473-3959-4e16-af06-0133607bc80d","Type":"ContainerStarted","Data":"c09cda1a7fde8b8fd194e306da345871ef2d5fb8fedfa2fe854c4bd1c915c987"} Nov 01 23:14:14.574628 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:14.574232 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lsr22" event={"ID":"ccca4473-3959-4e16-af06-0133607bc80d","Type":"ContainerStarted","Data":"08b5f32c69eda5ebe68efa207cd87659be76c7016081f5300f189fd89ae2ff54"} Nov 01 23:14:14.589947 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:14.589884 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lsr22" podStartSLOduration=1.589867393 podStartE2EDuration="1.589867393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:14:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:14:14.589646415 +0000 UTC m=+3022.994237051" watchObservedRunningTime="2026-11-01 23:14:14.589867393 +0000 UTC m=+3022.994458026" Nov 01 23:14:15.579899 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:15.579859 5011 generic.go:356] "Generic (PLEG): container finished" podID="ccca4473-3959-4e16-af06-0133607bc80d" containerID="c09cda1a7fde8b8fd194e306da345871ef2d5fb8fedfa2fe854c4bd1c915c987" exitCode=0 Nov 01 23:14:15.580315 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:15.579905 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lsr22" event={"ID":"ccca4473-3959-4e16-af06-0133607bc80d","Type":"ContainerDied","Data":"c09cda1a7fde8b8fd194e306da345871ef2d5fb8fedfa2fe854c4bd1c915c987"} Nov 01 23:14:16.724875 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:16.724845 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lsr22" Nov 01 23:14:16.875012 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:16.874916 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccca4473-3959-4e16-af06-0133607bc80d-kube-api-access-ctswt\" (UniqueName: \"kubernetes.io/projected/ccca4473-3959-4e16-af06-0133607bc80d-kube-api-access-ctswt\") pod \"ccca4473-3959-4e16-af06-0133607bc80d\" (UID: \"ccca4473-3959-4e16-af06-0133607bc80d\") " Nov 01 23:14:16.875012 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:16.874995 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccca4473-3959-4e16-af06-0133607bc80d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccca4473-3959-4e16-af06-0133607bc80d-maas-api-xks-ca\") pod \"ccca4473-3959-4e16-af06-0133607bc80d\" (UID: \"ccca4473-3959-4e16-af06-0133607bc80d\") " Nov 01 23:14:16.875261 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:16.875049 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccca4473-3959-4e16-af06-0133607bc80d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccca4473-3959-4e16-af06-0133607bc80d-maas-api-openshift-ca\") pod \"ccca4473-3959-4e16-af06-0133607bc80d\" (UID: \"ccca4473-3959-4e16-af06-0133607bc80d\") " Nov 01 23:14:16.875730 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:16.875454 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccca4473-3959-4e16-af06-0133607bc80d-maas-api-openshift-ca" pod "ccca4473-3959-4e16-af06-0133607bc80d" (UID: "ccca4473-3959-4e16-af06-0133607bc80d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:14:16.875730 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:16.875605 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccca4473-3959-4e16-af06-0133607bc80d-maas-api-openshift-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:14:16.877320 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:16.877295 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccca4473-3959-4e16-af06-0133607bc80d-kube-api-access-ctswt" pod "ccca4473-3959-4e16-af06-0133607bc80d" (UID: "ccca4473-3959-4e16-af06-0133607bc80d"). InnerVolumeSpecName "kube-api-access-ctswt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:14:16.877416 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:16.877356 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccca4473-3959-4e16-af06-0133607bc80d-maas-api-xks-ca" pod "ccca4473-3959-4e16-af06-0133607bc80d" (UID: "ccca4473-3959-4e16-af06-0133607bc80d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:14:16.958768 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:14:16.958732 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:14:16.976307 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:16.976279 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctswt\" (UniqueName: \"kubernetes.io/projected/ccca4473-3959-4e16-af06-0133607bc80d-kube-api-access-ctswt\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:14:16.976307 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:16.976305 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccca4473-3959-4e16-af06-0133607bc80d-maas-api-xks-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:14:17.590823 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:17.590788 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lsr22" event={"ID":"ccca4473-3959-4e16-af06-0133607bc80d","Type":"ContainerDied","Data":"08b5f32c69eda5ebe68efa207cd87659be76c7016081f5300f189fd89ae2ff54"} Nov 01 23:14:17.591013 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:17.590852 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08b5f32c69eda5ebe68efa207cd87659be76c7016081f5300f189fd89ae2ff54" Nov 01 23:14:17.591013 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:17.590854 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lsr22" Nov 01 23:14:17.634051 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:14:17.634017 5011 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccca4473_3959_4e16_af06_0133607bc80d.slice\": RecentStats: unable to find data in memory cache]" Nov 01 23:14:20.543553 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.543515 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr"] Nov 01 23:14:20.544149 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.543925 5011 cpu_manager.go:414] "containerMap: removing container" podUID="ccca4473-3959-4e16-af06-0133607bc80d" containerName="revoke-keys" Nov 01 23:14:20.544149 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.543936 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccca4473-3959-4e16-af06-0133607bc80d" containerName="revoke-keys" Nov 01 23:14:20.544149 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.544034 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccca4473-3959-4e16-af06-0133607bc80d" containerName="revoke-keys" Nov 01 23:14:20.547518 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.547499 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:20.559906 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.559878 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr"] Nov 01 23:14:20.612822 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.612786 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd3a9407-a464-4a77-b83a-b0755b5e0409-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-dxxsr\" (UID: \"dd3a9407-a464-4a77-b83a-b0755b5e0409\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:20.612822 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.612826 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd3a9407-a464-4a77-b83a-b0755b5e0409-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-dxxsr\" (UID: \"dd3a9407-a464-4a77-b83a-b0755b5e0409\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:20.613070 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.612925 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd3a9407-a464-4a77-b83a-b0755b5e0409-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-dxxsr\" (UID: \"dd3a9407-a464-4a77-b83a-b0755b5e0409\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:20.613070 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.612976 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd3a9407-a464-4a77-b83a-b0755b5e0409-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-dxxsr\" (UID: \"dd3a9407-a464-4a77-b83a-b0755b5e0409\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:20.613161 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.613071 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd3a9407-a464-4a77-b83a-b0755b5e0409-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-dxxsr\" (UID: \"dd3a9407-a464-4a77-b83a-b0755b5e0409\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:20.613161 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.613105 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd3a9407-a464-4a77-b83a-b0755b5e0409-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-dxxsr\" (UID: \"dd3a9407-a464-4a77-b83a-b0755b5e0409\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:20.613161 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.613132 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd3a9407-a464-4a77-b83a-b0755b5e0409-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-dxxsr\" (UID: \"dd3a9407-a464-4a77-b83a-b0755b5e0409\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:20.613280 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.613193 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd3a9407-a464-4a77-b83a-b0755b5e0409-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-dxxsr\" (UID: \"dd3a9407-a464-4a77-b83a-b0755b5e0409\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:20.613280 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.613210 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh4xk\" (UniqueName: \"kubernetes.io/projected/dd3a9407-a464-4a77-b83a-b0755b5e0409-kube-api-access-fh4xk\") pod \"maas-default-gateway-openshift-default-6886b96f9-dxxsr\" (UID: \"dd3a9407-a464-4a77-b83a-b0755b5e0409\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:20.714594 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.714551 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd3a9407-a464-4a77-b83a-b0755b5e0409-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-dxxsr\" (UID: \"dd3a9407-a464-4a77-b83a-b0755b5e0409\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:20.714797 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.714605 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd3a9407-a464-4a77-b83a-b0755b5e0409-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-dxxsr\" (UID: \"dd3a9407-a464-4a77-b83a-b0755b5e0409\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:20.714797 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.714635 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd3a9407-a464-4a77-b83a-b0755b5e0409-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-dxxsr\" (UID: \"dd3a9407-a464-4a77-b83a-b0755b5e0409\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:20.714797 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.714678 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd3a9407-a464-4a77-b83a-b0755b5e0409-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-dxxsr\" (UID: \"dd3a9407-a464-4a77-b83a-b0755b5e0409\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:20.714797 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.714739 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd3a9407-a464-4a77-b83a-b0755b5e0409-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-dxxsr\" (UID: \"dd3a9407-a464-4a77-b83a-b0755b5e0409\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:20.714797 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.714775 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd3a9407-a464-4a77-b83a-b0755b5e0409-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-dxxsr\" (UID: \"dd3a9407-a464-4a77-b83a-b0755b5e0409\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:20.715134 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.714813 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd3a9407-a464-4a77-b83a-b0755b5e0409-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-dxxsr\" (UID: \"dd3a9407-a464-4a77-b83a-b0755b5e0409\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:20.715134 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.714922 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd3a9407-a464-4a77-b83a-b0755b5e0409-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-dxxsr\" (UID: \"dd3a9407-a464-4a77-b83a-b0755b5e0409\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:20.715134 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.714948 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh4xk\" (UniqueName: \"kubernetes.io/projected/dd3a9407-a464-4a77-b83a-b0755b5e0409-kube-api-access-fh4xk\") pod \"maas-default-gateway-openshift-default-6886b96f9-dxxsr\" (UID: \"dd3a9407-a464-4a77-b83a-b0755b5e0409\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:20.715134 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.715082 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd3a9407-a464-4a77-b83a-b0755b5e0409-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-dxxsr\" (UID: \"dd3a9407-a464-4a77-b83a-b0755b5e0409\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:20.715336 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.715188 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd3a9407-a464-4a77-b83a-b0755b5e0409-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-dxxsr\" (UID: \"dd3a9407-a464-4a77-b83a-b0755b5e0409\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:20.715336 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.715233 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd3a9407-a464-4a77-b83a-b0755b5e0409-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-dxxsr\" (UID: \"dd3a9407-a464-4a77-b83a-b0755b5e0409\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:20.715421 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.715396 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd3a9407-a464-4a77-b83a-b0755b5e0409-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-dxxsr\" (UID: \"dd3a9407-a464-4a77-b83a-b0755b5e0409\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:20.715577 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.715556 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd3a9407-a464-4a77-b83a-b0755b5e0409-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-dxxsr\" (UID: \"dd3a9407-a464-4a77-b83a-b0755b5e0409\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:20.717437 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.717411 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd3a9407-a464-4a77-b83a-b0755b5e0409-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-dxxsr\" (UID: \"dd3a9407-a464-4a77-b83a-b0755b5e0409\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:20.717615 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.717599 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd3a9407-a464-4a77-b83a-b0755b5e0409-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-dxxsr\" (UID: \"dd3a9407-a464-4a77-b83a-b0755b5e0409\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:20.722446 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.722425 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd3a9407-a464-4a77-b83a-b0755b5e0409-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-dxxsr\" (UID: \"dd3a9407-a464-4a77-b83a-b0755b5e0409\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:20.722645 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.722624 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh4xk\" (UniqueName: \"kubernetes.io/projected/dd3a9407-a464-4a77-b83a-b0755b5e0409-kube-api-access-fh4xk\") pod \"maas-default-gateway-openshift-default-6886b96f9-dxxsr\" (UID: \"dd3a9407-a464-4a77-b83a-b0755b5e0409\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:20.861335 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:20.861232 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:21.138998 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:21.138928 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr"] Nov 01 23:14:21.141244 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:14:21.141213 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd3a9407_a464_4a77_b83a_b0755b5e0409.slice/crio-66112cb29d7c916c46b4aaae498e238011902792234fa427a76aeb22a1d057e4 WatchSource:0}: Error finding container 66112cb29d7c916c46b4aaae498e238011902792234fa427a76aeb22a1d057e4: Status 404 returned error can't find the container with id 66112cb29d7c916c46b4aaae498e238011902792234fa427a76aeb22a1d057e4 Nov 01 23:14:21.143391 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:21.143354 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:14:21.143474 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:21.143435 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:14:21.143474 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:21.143464 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:14:21.613804 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:21.613769 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" event={"ID":"dd3a9407-a464-4a77-b83a-b0755b5e0409","Type":"ContainerStarted","Data":"8372a9eccbcc3e10b397b22e1f7f8b84c125650f1b5fea54ba38de4529807b70"} Nov 01 23:14:21.614248 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:21.613824 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" event={"ID":"dd3a9407-a464-4a77-b83a-b0755b5e0409","Type":"ContainerStarted","Data":"66112cb29d7c916c46b4aaae498e238011902792234fa427a76aeb22a1d057e4"} Nov 01 23:14:21.636015 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:21.635963 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" podStartSLOduration=1.635948663 podStartE2EDuration="1.635948663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:14:21.634089454 +0000 UTC m=+3030.038680087" watchObservedRunningTime="2026-11-01 23:14:21.635948663 +0000 UTC m=+3030.040539294" Nov 01 23:14:21.865913 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:21.862925 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:22.866400 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:22.866359 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dxxsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Nov 01 23:14:22.866955 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:22.866423 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" podUID="dd3a9407-a464-4a77-b83a-b0755b5e0409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Nov 01 23:14:23.862856 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:23.862256 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dxxsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Nov 01 23:14:23.862856 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:23.862319 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" podUID="dd3a9407-a464-4a77-b83a-b0755b5e0409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Nov 01 23:14:24.385857 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:24.385204 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:14:24.864093 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:24.864056 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dxxsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Nov 01 23:14:24.864417 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:24.864372 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" podUID="dd3a9407-a464-4a77-b83a-b0755b5e0409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Nov 01 23:14:25.865268 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:25.865228 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dxxsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Nov 01 23:14:25.865921 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:25.865888 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" podUID="dd3a9407-a464-4a77-b83a-b0755b5e0409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Nov 01 23:14:26.865857 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:26.865255 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dxxsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Nov 01 23:14:26.865857 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:26.865321 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" podUID="dd3a9407-a464-4a77-b83a-b0755b5e0409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Nov 01 23:14:27.759548 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:27.759516 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:14:27.861997 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:27.861963 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dxxsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Nov 01 23:14:27.862197 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:27.862029 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" podUID="dd3a9407-a464-4a77-b83a-b0755b5e0409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Nov 01 23:14:29.872614 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:29.871951 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dxxsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:14:29.872614 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:29.872038 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" podUID="dd3a9407-a464-4a77-b83a-b0755b5e0409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:14:30.372048 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:30.372003 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-m4nvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:14:30.372412 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:30.372379 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" podUID="fd9832d7-c187-41b2-bf9d-de25b0ea455a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:14:30.873150 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:30.873104 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dxxsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:14:30.873701 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:30.873178 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" podUID="dd3a9407-a464-4a77-b83a-b0755b5e0409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:14:31.914022 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:31.913985 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dxxsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:14:31.914431 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:31.914085 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" podUID="dd3a9407-a464-4a77-b83a-b0755b5e0409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:14:32.914748 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:32.914707 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dxxsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:14:32.915211 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:32.914793 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" podUID="dd3a9407-a464-4a77-b83a-b0755b5e0409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:14:33.956005 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:33.955965 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dxxsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:14:33.956636 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:33.956585 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" podUID="dd3a9407-a464-4a77-b83a-b0755b5e0409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:14:34.100111 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:34.100070 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:34.100383 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:34.100361 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:34.101249 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:34.101231 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" Nov 01 23:14:34.202173 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:34.202141 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wjs4b"] Nov 01 23:14:34.206147 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:34.206082 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wjs4b" Nov 01 23:14:34.212670 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:34.212224 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:14:34.212670 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:34.212315 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:14:34.212670 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:34.212362 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:14:34.212670 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:34.212433 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:14:34.217485 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:34.217455 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wjs4b"] Nov 01 23:14:34.287779 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:34.287740 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-wjs4b\" (UID: \"a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wjs4b" Nov 01 23:14:34.288034 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:34.287890 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhkfm\" (UniqueName: \"kubernetes.io/projected/a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7-kube-api-access-lhkfm\") pod \"maas-api-revoke-sub-2e1f5b5f-wjs4b\" (UID: \"a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wjs4b" Nov 01 23:14:34.288034 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:34.287978 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-wjs4b\" (UID: \"a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wjs4b" Nov 01 23:14:34.389018 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:34.388975 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-wjs4b\" (UID: \"a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wjs4b" Nov 01 23:14:34.389200 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:34.389054 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhkfm\" (UniqueName: \"kubernetes.io/projected/a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7-kube-api-access-lhkfm\") pod \"maas-api-revoke-sub-2e1f5b5f-wjs4b\" (UID: \"a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wjs4b" Nov 01 23:14:34.389200 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:34.389114 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-wjs4b\" (UID: \"a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wjs4b" Nov 01 23:14:34.389728 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:34.389704 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-wjs4b\" (UID: \"a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wjs4b" Nov 01 23:14:34.391598 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:34.391571 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-wjs4b\" (UID: \"a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wjs4b" Nov 01 23:14:34.396814 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:34.396794 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhkfm\" (UniqueName: \"kubernetes.io/projected/a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7-kube-api-access-lhkfm\") pod \"maas-api-revoke-sub-2e1f5b5f-wjs4b\" (UID: \"a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wjs4b" Nov 01 23:14:34.525798 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:34.525713 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wjs4b" Nov 01 23:14:34.799489 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:34.799462 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wjs4b"] Nov 01 23:14:34.801634 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:14:34.801608 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda11fe6f0_3ef5_4b91_8c9d_75af1561bbb7.slice/crio-9e936afab54319159e40be556cf2b557fe369da9b1a2a46a2c2986cd433b9fc3 WatchSource:0}: Error finding container 9e936afab54319159e40be556cf2b557fe369da9b1a2a46a2c2986cd433b9fc3: Status 404 returned error can't find the container with id 9e936afab54319159e40be556cf2b557fe369da9b1a2a46a2c2986cd433b9fc3 Nov 01 23:14:35.706751 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:35.706712 5011 generic.go:356] "Generic (PLEG): container finished" podID="a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7" containerID="9ce07da4fb39d3def0352d09270b13764508641d23632a0796d5b02fa728aff1" exitCode=0 Nov 01 23:14:35.707136 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:35.706799 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wjs4b" event={"ID":"a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7","Type":"ContainerDied","Data":"9ce07da4fb39d3def0352d09270b13764508641d23632a0796d5b02fa728aff1"} Nov 01 23:14:35.707136 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:35.706865 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wjs4b" event={"ID":"a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7","Type":"ContainerStarted","Data":"9e936afab54319159e40be556cf2b557fe369da9b1a2a46a2c2986cd433b9fc3"} Nov 01 23:14:36.848966 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:36.848936 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wjs4b" Nov 01 23:14:36.916648 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:36.916614 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7-maas-api-openshift-ca\") pod \"a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7\" (UID: \"a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7\") " Nov 01 23:14:36.916824 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:36.916655 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7-maas-api-xks-ca\") pod \"a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7\" (UID: \"a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7\") " Nov 01 23:14:36.916824 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:36.916714 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7-kube-api-access-lhkfm\" (UniqueName: \"kubernetes.io/projected/a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7-kube-api-access-lhkfm\") pod \"a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7\" (UID: \"a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7\") " Nov 01 23:14:36.917078 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:36.917051 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7-maas-api-openshift-ca" pod "a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7" (UID: "a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:14:36.919039 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:36.919014 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7-maas-api-xks-ca" pod "a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7" (UID: "a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:14:36.919039 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:36.919023 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7-kube-api-access-lhkfm" pod "a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7" (UID: "a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7"). InnerVolumeSpecName "kube-api-access-lhkfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:14:37.018499 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:37.018415 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhkfm\" (UniqueName: \"kubernetes.io/projected/a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7-kube-api-access-lhkfm\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:14:37.018499 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:37.018449 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7-maas-api-openshift-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:14:37.018499 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:37.018466 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7-maas-api-xks-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:14:37.718063 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:37.718035 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wjs4b" Nov 01 23:14:37.718063 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:37.718053 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wjs4b" event={"ID":"a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7","Type":"ContainerDied","Data":"9e936afab54319159e40be556cf2b557fe369da9b1a2a46a2c2986cd433b9fc3"} Nov 01 23:14:37.718290 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:37.718096 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e936afab54319159e40be556cf2b557fe369da9b1a2a46a2c2986cd433b9fc3" Nov 01 23:14:45.086386 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:45.086347 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:14:46.959317 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:14:46.959281 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:14:50.149038 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:50.148993 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dxxsr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:14:50.149716 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:50.149679 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dxxsr" podUID="dd3a9407-a464-4a77-b83a-b0755b5e0409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:14:58.002022 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:14:58.001985 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:15:00.404871 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:00.374939 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-m4nvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:15:00.404871 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:00.375030 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" podUID="fd9832d7-c187-41b2-bf9d-de25b0ea455a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:15:11.437981 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:11.437939 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7sb5k"] Nov 01 23:15:11.438385 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:11.438341 5011 cpu_manager.go:414] "containerMap: removing container" podUID="a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7" containerName="revoke-keys" Nov 01 23:15:11.438385 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:11.438353 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7" containerName="revoke-keys" Nov 01 23:15:11.438469 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:11.438419 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7" containerName="revoke-keys" Nov 01 23:15:11.442767 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:11.442750 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7sb5k" Nov 01 23:15:11.446367 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:11.446268 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:15:11.447445 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:11.447404 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:15:11.447445 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:11.447424 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:15:11.447669 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:11.447519 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:15:11.449861 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:11.449801 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7sb5k"] Nov 01 23:15:11.479766 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:11.479737 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-7sb5k\" (UID: \"4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7sb5k" Nov 01 23:15:11.479949 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:11.479796 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wg5m\" (UniqueName: \"kubernetes.io/projected/4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca-kube-api-access-2wg5m\") pod \"maas-api-revoke-sub-046e1343-7sb5k\" (UID: \"4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7sb5k" Nov 01 23:15:11.480003 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:11.479968 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-7sb5k\" (UID: \"4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7sb5k" Nov 01 23:15:11.580571 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:11.580534 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-7sb5k\" (UID: \"4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7sb5k" Nov 01 23:15:11.580754 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:11.580591 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wg5m\" (UniqueName: \"kubernetes.io/projected/4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca-kube-api-access-2wg5m\") pod \"maas-api-revoke-sub-046e1343-7sb5k\" (UID: \"4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7sb5k" Nov 01 23:15:11.580754 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:11.580657 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-7sb5k\" (UID: \"4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7sb5k" Nov 01 23:15:11.581184 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:11.581163 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-7sb5k\" (UID: \"4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7sb5k" Nov 01 23:15:11.583142 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:11.583121 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-7sb5k\" (UID: \"4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7sb5k" Nov 01 23:15:11.588232 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:11.588211 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wg5m\" (UniqueName: \"kubernetes.io/projected/4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca-kube-api-access-2wg5m\") pod \"maas-api-revoke-sub-046e1343-7sb5k\" (UID: \"4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7sb5k" Nov 01 23:15:11.757986 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:11.757826 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7sb5k" Nov 01 23:15:12.031430 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:12.031402 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7sb5k"] Nov 01 23:15:12.033525 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:15:12.033494 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c3ddb2d_fe16_42e1_95c8_d3b8d3e79eca.slice/crio-364a86c5bea9ea0953ab6d47f4d7ac1347504a0fbc2cfc148c9df0b55cd42399 WatchSource:0}: Error finding container 364a86c5bea9ea0953ab6d47f4d7ac1347504a0fbc2cfc148c9df0b55cd42399: Status 404 returned error can't find the container with id 364a86c5bea9ea0953ab6d47f4d7ac1347504a0fbc2cfc148c9df0b55cd42399 Nov 01 23:15:12.922181 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:12.922142 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7sb5k" event={"ID":"4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca","Type":"ContainerStarted","Data":"6b7861f843cd30b340f9e21c4ec56477c09b1c7b02196467a49f89897cb4684e"} Nov 01 23:15:12.922594 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:12.922210 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7sb5k" event={"ID":"4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca","Type":"ContainerStarted","Data":"364a86c5bea9ea0953ab6d47f4d7ac1347504a0fbc2cfc148c9df0b55cd42399"} Nov 01 23:15:12.942286 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:12.942239 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7sb5k" podStartSLOduration=1.942224068 podStartE2EDuration="1.942224068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:15:12.939756062 +0000 UTC m=+3081.344346694" watchObservedRunningTime="2026-11-01 23:15:12.942224068 +0000 UTC m=+3081.346814701" Nov 01 23:15:13.928212 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:13.928176 5011 generic.go:356] "Generic (PLEG): container finished" podID="4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca" containerID="6b7861f843cd30b340f9e21c4ec56477c09b1c7b02196467a49f89897cb4684e" exitCode=0 Nov 01 23:15:13.928619 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:13.928258 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7sb5k" event={"ID":"4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca","Type":"ContainerDied","Data":"6b7861f843cd30b340f9e21c4ec56477c09b1c7b02196467a49f89897cb4684e"} Nov 01 23:15:15.067575 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:15.067552 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7sb5k" Nov 01 23:15:15.116481 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:15.116435 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca-maas-api-xks-ca\") pod \"4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca\" (UID: \"4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca\") " Nov 01 23:15:15.116686 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:15.116501 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca-maas-api-openshift-ca\") pod \"4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca\" (UID: \"4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca\") " Nov 01 23:15:15.116686 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:15.116642 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca-kube-api-access-2wg5m\" (UniqueName: \"kubernetes.io/projected/4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca-kube-api-access-2wg5m\") pod \"4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca\" (UID: \"4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca\") " Nov 01 23:15:15.116882 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:15.116858 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca-maas-api-openshift-ca" pod "4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca" (UID: "4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:15:15.117066 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:15.117042 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca-maas-api-openshift-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:15:15.118845 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:15.118803 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca-maas-api-xks-ca" pod "4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca" (UID: "4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:15:15.118964 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:15.118939 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca-kube-api-access-2wg5m" pod "4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca" (UID: "4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca"). InnerVolumeSpecName "kube-api-access-2wg5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:15:15.217590 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:15.217508 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wg5m\" (UniqueName: \"kubernetes.io/projected/4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca-kube-api-access-2wg5m\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:15:15.217590 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:15.217536 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca-maas-api-xks-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:15:15.939600 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:15.939571 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7sb5k" Nov 01 23:15:15.939778 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:15.939571 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7sb5k" event={"ID":"4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca","Type":"ContainerDied","Data":"364a86c5bea9ea0953ab6d47f4d7ac1347504a0fbc2cfc148c9df0b55cd42399"} Nov 01 23:15:15.939778 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:15.939698 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="364a86c5bea9ea0953ab6d47f4d7ac1347504a0fbc2cfc148c9df0b55cd42399" Nov 01 23:15:16.959241 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:15:16.959195 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:15:22.347925 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:22.347892 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:15:32.521691 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:32.521652 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:15:36.216978 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:36.216945 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:15:46.959308 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:15:46.959266 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:15:52.380414 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:52.380377 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:15:55.916297 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:15:55.916260 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:16:10.289071 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:16:10.289036 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:16:14.700858 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:16:14.700012 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-z9x5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:16:14.700858 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:16:14.700101 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" podUID="734444e5-d903-47e2-9ed0-d7617fd3b2ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:16:15.357455 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:16:15.357415 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:16:16.958470 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:16:16.958434 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:16:28.809570 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:16:28.809530 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:16:32.536536 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:16:32.536314 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:16:46.516519 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:16:46.516480 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:16:46.973249 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:16:46.973208 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:16:57.184355 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:16:57.184317 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:16:59.695857 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:16:59.694985 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-z9x5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:16:59.695857 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:16:59.695082 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" podUID="734444e5-d903-47e2-9ed0-d7617fd3b2ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:17:01.677859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:17:01.674403 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:17:02.429018 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:17:02.428972 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bxw6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:17:02.429405 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:17:02.429371 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" podUID="4688829c-8b38-4d0f-a56c-f20892644b09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:17:13.032792 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:17:13.032747 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-639c4fc8-6srhf"] Nov 01 23:17:13.035011 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:17:13.034985 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-639c4fc8-6srhf"] Nov 01 23:17:13.745960 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:17:13.745926 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86aac669-ecb7-42be-9a9f-37d59ee94e5d" path="/var/lib/kubelet/pods/86aac669-ecb7-42be-9a9f-37d59ee94e5d/volumes" Nov 01 23:17:15.708620 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:17:15.708589 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:17:16.959503 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:17:16.959468 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:17:19.300190 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:17:19.300155 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:17:25.275054 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:17:25.275018 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:17:41.031518 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:17:41.031478 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4517e7a2-d2vvv"] Nov 01 23:17:41.033657 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:17:41.033630 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4517e7a2-d2vvv"] Nov 01 23:17:41.745433 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:17:41.745399 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80493f88-4b1b-4d2c-acb5-f956a719af86" path="/var/lib/kubelet/pods/80493f88-4b1b-4d2c-acb5-f956a719af86/volumes" Nov 01 23:17:43.395547 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:17:43.395513 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:17:46.960206 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:17:46.960167 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:17:53.673861 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:17:53.673805 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:18:09.208716 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:18:09.208631 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:18:13.494770 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:18:13.494735 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:18:16.958901 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:18:16.958860 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:18:20.586966 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:18:20.586929 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-aaf3e1-5f577mnw6k"] Nov 01 23:18:20.594130 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:18:20.594106 5011 cpu_manager.go:414] "containerMap: removing container" podUID="4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca" containerName="revoke-keys" Nov 01 23:18:20.594130 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:18:20.594133 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca" containerName="revoke-keys" Nov 01 23:18:20.594275 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:18:20.594265 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca" containerName="revoke-keys" Nov 01 23:18:20.598505 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:18:20.598477 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aaf3e1-5f577mnw6k" Nov 01 23:18:20.600969 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:18:20.600940 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-aaf3e1-5f577mnw6k"] Nov 01 23:18:20.602025 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:18:20.602004 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-aaf3e1\"" Nov 01 23:18:20.603010 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:18:20.602988 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-94d6703a-dockercfg-9pgtk\"" Nov 01 23:18:20.708137 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:18:20.707885 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3633b776-74ea-4214-9f8d-627a695904cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-aaf3e1-5f577mnw6k\" (UID: \"3633b776-74ea-4214-9f8d-627a695904cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aaf3e1-5f577mnw6k" Nov 01 23:18:20.708137 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:18:20.708000 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hsgr\" (UniqueName: \"kubernetes.io/projected/3633b776-74ea-4214-9f8d-627a695904cb-kube-api-access-6hsgr\") pod \"payload-pre-processing-e2e-worker-single-main-aaf3e1-5f577mnw6k\" (UID: \"3633b776-74ea-4214-9f8d-627a695904cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aaf3e1-5f577mnw6k" Nov 01 23:18:20.809349 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:18:20.809305 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3633b776-74ea-4214-9f8d-627a695904cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-aaf3e1-5f577mnw6k\" (UID: \"3633b776-74ea-4214-9f8d-627a695904cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aaf3e1-5f577mnw6k" Nov 01 23:18:20.809349 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:18:20.809353 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hsgr\" (UniqueName: \"kubernetes.io/projected/3633b776-74ea-4214-9f8d-627a695904cb-kube-api-access-6hsgr\") pod \"payload-pre-processing-e2e-worker-single-main-aaf3e1-5f577mnw6k\" (UID: \"3633b776-74ea-4214-9f8d-627a695904cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aaf3e1-5f577mnw6k" Nov 01 23:18:20.810016 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:18:20.809996 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3633b776-74ea-4214-9f8d-627a695904cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-aaf3e1-5f577mnw6k\" (UID: \"3633b776-74ea-4214-9f8d-627a695904cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aaf3e1-5f577mnw6k" Nov 01 23:18:20.817462 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:18:20.817432 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hsgr\" (UniqueName: \"kubernetes.io/projected/3633b776-74ea-4214-9f8d-627a695904cb-kube-api-access-6hsgr\") pod \"payload-pre-processing-e2e-worker-single-main-aaf3e1-5f577mnw6k\" (UID: \"3633b776-74ea-4214-9f8d-627a695904cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aaf3e1-5f577mnw6k" Nov 01 23:18:20.915308 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:18:20.915215 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aaf3e1-5f577mnw6k" Nov 01 23:18:21.217188 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:18:21.217158 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-aaf3e1-5f577mnw6k"] Nov 01 23:18:21.221138 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:18:21.221099 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3633b776_74ea_4214_9f8d_627a695904cb.slice/crio-c39f9bbefd439437037661f798392ec0298c99daf04c3788a95d9bd7cb979ed3 WatchSource:0}: Error finding container c39f9bbefd439437037661f798392ec0298c99daf04c3788a95d9bd7cb979ed3: Status 404 returned error can't find the container with id c39f9bbefd439437037661f798392ec0298c99daf04c3788a95d9bd7cb979ed3 Nov 01 23:18:22.144916 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:18:22.144876 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aaf3e1-5f577mnw6k" event={"ID":"3633b776-74ea-4214-9f8d-627a695904cb","Type":"ContainerStarted","Data":"a3418dcf7f91cdcb453f0bebfcb92f4346fab9e229c143e1e30b2ca286f2f080"} Nov 01 23:18:22.145334 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:18:22.144938 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aaf3e1-5f577mnw6k" event={"ID":"3633b776-74ea-4214-9f8d-627a695904cb","Type":"ContainerStarted","Data":"c39f9bbefd439437037661f798392ec0298c99daf04c3788a95d9bd7cb979ed3"} Nov 01 23:18:22.145334 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:18:22.145056 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aaf3e1-5f577mnw6k" Nov 01 23:18:22.161413 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:18:22.161367 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aaf3e1-5f577mnw6k" podStartSLOduration=2.161354889 podStartE2EDuration="2.161354889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:18:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:18:22.160174619 +0000 UTC m=+3270.564765255" watchObservedRunningTime="2026-11-01 23:18:22.161354889 +0000 UTC m=+3270.565945521" Nov 01 23:18:22.292904 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:18:22.292866 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:18:25.019677 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:18:25.019146 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gk9kf"] Nov 01 23:18:25.022249 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:18:25.022221 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-gk9kf"] Nov 01 23:18:25.745122 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:18:25.745092 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6078069-ba77-49f3-bf44-58db4e58e202" path="/var/lib/kubelet/pods/a6078069-ba77-49f3-bf44-58db4e58e202/volumes" Nov 01 23:18:33.153666 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:18:33.153633 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aaf3e1-5f577mnw6k" Nov 01 23:18:39.501858 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:18:39.501791 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:18:46.960197 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:18:46.960153 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:18:54.509682 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:18:54.509644 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:18:55.702147 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:18:55.702109 5011 scope.go:122] "RemoveContainer" containerID="9c26287230dbf91e213ac2715a713c326620bd971135da525ee2de1afb471aa0" Nov 01 23:18:55.712140 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:18:55.712117 5011 scope.go:122] "RemoveContainer" containerID="daee16e3ed3a1cde2226dcc632208a3de02f6904826c7b8267d38727d16c2d84" Nov 01 23:19:05.946871 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:05.946820 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65ac655d-7r4mc"] Nov 01 23:19:05.951771 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:05.951753 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ac655d-7r4mc" Nov 01 23:19:05.956023 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:05.955995 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:19:05.956654 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:05.956631 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:19:05.956919 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:05.956898 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:19:05.957044 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:05.957021 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:19:05.957927 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:05.957861 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65ac655d-7r4mc"] Nov 01 23:19:06.007362 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:06.007329 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e573717-83e3-4002-a886-942b595c88a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65ac655d-7r4mc\" (UID: \"1e573717-83e3-4002-a886-942b595c88a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ac655d-7r4mc" Nov 01 23:19:06.007556 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:06.007385 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b44r\" (UniqueName: \"kubernetes.io/projected/1e573717-83e3-4002-a886-942b595c88a1-kube-api-access-8b44r\") pod \"maas-api-revoke-sub-65ac655d-7r4mc\" (UID: \"1e573717-83e3-4002-a886-942b595c88a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ac655d-7r4mc" Nov 01 23:19:06.007556 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:06.007476 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e573717-83e3-4002-a886-942b595c88a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65ac655d-7r4mc\" (UID: \"1e573717-83e3-4002-a886-942b595c88a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ac655d-7r4mc" Nov 01 23:19:06.108358 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:06.108323 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b44r\" (UniqueName: \"kubernetes.io/projected/1e573717-83e3-4002-a886-942b595c88a1-kube-api-access-8b44r\") pod \"maas-api-revoke-sub-65ac655d-7r4mc\" (UID: \"1e573717-83e3-4002-a886-942b595c88a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ac655d-7r4mc" Nov 01 23:19:06.108358 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:06.108362 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e573717-83e3-4002-a886-942b595c88a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65ac655d-7r4mc\" (UID: \"1e573717-83e3-4002-a886-942b595c88a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ac655d-7r4mc" Nov 01 23:19:06.108619 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:06.108421 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e573717-83e3-4002-a886-942b595c88a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65ac655d-7r4mc\" (UID: \"1e573717-83e3-4002-a886-942b595c88a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ac655d-7r4mc" Nov 01 23:19:06.109069 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:06.109047 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e573717-83e3-4002-a886-942b595c88a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65ac655d-7r4mc\" (UID: \"1e573717-83e3-4002-a886-942b595c88a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ac655d-7r4mc" Nov 01 23:19:06.111140 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:06.111115 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e573717-83e3-4002-a886-942b595c88a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65ac655d-7r4mc\" (UID: \"1e573717-83e3-4002-a886-942b595c88a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ac655d-7r4mc" Nov 01 23:19:06.116616 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:06.116591 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b44r\" (UniqueName: \"kubernetes.io/projected/1e573717-83e3-4002-a886-942b595c88a1-kube-api-access-8b44r\") pod \"maas-api-revoke-sub-65ac655d-7r4mc\" (UID: \"1e573717-83e3-4002-a886-942b595c88a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ac655d-7r4mc" Nov 01 23:19:06.268649 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:06.268568 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ac655d-7r4mc" Nov 01 23:19:06.543150 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:06.543126 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65ac655d-7r4mc"] Nov 01 23:19:06.545756 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:19:06.545720 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e573717_83e3_4002_a886_942b595c88a1.slice/crio-bafdf72b66d968399a256a81629f84e227443a85930677f420fef3ca8c7e8359 WatchSource:0}: Error finding container bafdf72b66d968399a256a81629f84e227443a85930677f420fef3ca8c7e8359: Status 404 returned error can't find the container with id bafdf72b66d968399a256a81629f84e227443a85930677f420fef3ca8c7e8359 Nov 01 23:19:07.400404 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:07.400368 5011 generic.go:356] "Generic (PLEG): container finished" podID="1e573717-83e3-4002-a886-942b595c88a1" containerID="389b6e1027ec1e4c48776c7d5e4bbb9debc43a5fd5f4100f7991e504dace69db" exitCode=0 Nov 01 23:19:07.400804 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:07.400442 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ac655d-7r4mc" event={"ID":"1e573717-83e3-4002-a886-942b595c88a1","Type":"ContainerDied","Data":"389b6e1027ec1e4c48776c7d5e4bbb9debc43a5fd5f4100f7991e504dace69db"} Nov 01 23:19:07.400804 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:07.400505 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ac655d-7r4mc" event={"ID":"1e573717-83e3-4002-a886-942b595c88a1","Type":"ContainerStarted","Data":"bafdf72b66d968399a256a81629f84e227443a85930677f420fef3ca8c7e8359"} Nov 01 23:19:08.546103 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:08.546070 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ac655d-7r4mc" Nov 01 23:19:08.640259 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:08.640229 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e573717-83e3-4002-a886-942b595c88a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e573717-83e3-4002-a886-942b595c88a1-maas-api-openshift-ca\") pod \"1e573717-83e3-4002-a886-942b595c88a1\" (UID: \"1e573717-83e3-4002-a886-942b595c88a1\") " Nov 01 23:19:08.640453 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:08.640336 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e573717-83e3-4002-a886-942b595c88a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e573717-83e3-4002-a886-942b595c88a1-maas-api-xks-ca\") pod \"1e573717-83e3-4002-a886-942b595c88a1\" (UID: \"1e573717-83e3-4002-a886-942b595c88a1\") " Nov 01 23:19:08.640453 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:08.640367 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e573717-83e3-4002-a886-942b595c88a1-kube-api-access-8b44r\" (UniqueName: \"kubernetes.io/projected/1e573717-83e3-4002-a886-942b595c88a1-kube-api-access-8b44r\") pod \"1e573717-83e3-4002-a886-942b595c88a1\" (UID: \"1e573717-83e3-4002-a886-942b595c88a1\") " Nov 01 23:19:08.640607 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:08.640582 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e573717-83e3-4002-a886-942b595c88a1-maas-api-openshift-ca" pod "1e573717-83e3-4002-a886-942b595c88a1" (UID: "1e573717-83e3-4002-a886-942b595c88a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:19:08.642526 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:08.642494 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e573717-83e3-4002-a886-942b595c88a1-maas-api-xks-ca" pod "1e573717-83e3-4002-a886-942b595c88a1" (UID: "1e573717-83e3-4002-a886-942b595c88a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:19:08.642650 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:08.642558 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e573717-83e3-4002-a886-942b595c88a1-kube-api-access-8b44r" pod "1e573717-83e3-4002-a886-942b595c88a1" (UID: "1e573717-83e3-4002-a886-942b595c88a1"). InnerVolumeSpecName "kube-api-access-8b44r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:19:08.741086 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:08.741055 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e573717-83e3-4002-a886-942b595c88a1-maas-api-xks-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:08.741086 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:08.741084 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b44r\" (UniqueName: \"kubernetes.io/projected/1e573717-83e3-4002-a886-942b595c88a1-kube-api-access-8b44r\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:08.741306 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:08.741098 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e573717-83e3-4002-a886-942b595c88a1-maas-api-openshift-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:09.415114 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:09.415076 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ac655d-7r4mc" event={"ID":"1e573717-83e3-4002-a886-942b595c88a1","Type":"ContainerDied","Data":"bafdf72b66d968399a256a81629f84e227443a85930677f420fef3ca8c7e8359"} Nov 01 23:19:09.415287 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:09.415133 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bafdf72b66d968399a256a81629f84e227443a85930677f420fef3ca8c7e8359" Nov 01 23:19:09.415287 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:09.415093 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ac655d-7r4mc" Nov 01 23:19:16.930196 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:16.930160 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aaf3e1-psg9p"] Nov 01 23:19:16.930811 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:16.930750 5011 cpu_manager.go:414] "containerMap: removing container" podUID="1e573717-83e3-4002-a886-942b595c88a1" containerName="revoke-keys" Nov 01 23:19:16.930811 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:16.930771 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e573717-83e3-4002-a886-942b595c88a1" containerName="revoke-keys" Nov 01 23:19:16.930986 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:16.930928 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e573717-83e3-4002-a886-942b595c88a1" containerName="revoke-keys" Nov 01 23:19:16.934303 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:16.934279 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aaf3e1-psg9p" Nov 01 23:19:16.940023 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:16.939993 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:19:16.940174 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:16.940040 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:19:16.940174 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:16.940084 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:19:16.940174 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:16.940170 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:19:16.941685 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:16.941659 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aaf3e1-psg9p"] Nov 01 23:19:16.966077 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:19:16.966039 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:19:17.021794 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:17.021759 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks5rn\" (UniqueName: \"kubernetes.io/projected/79f34e4c-e182-467f-8d22-56389ad95d89-kube-api-access-ks5rn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-aaf3e1-psg9p\" (UID: \"79f34e4c-e182-467f-8d22-56389ad95d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aaf3e1-psg9p" Nov 01 23:19:17.022007 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:17.021815 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79f34e4c-e182-467f-8d22-56389ad95d89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-aaf3e1-psg9p\" (UID: \"79f34e4c-e182-467f-8d22-56389ad95d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aaf3e1-psg9p" Nov 01 23:19:17.022007 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:17.021957 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79f34e4c-e182-467f-8d22-56389ad95d89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-aaf3e1-psg9p\" (UID: \"79f34e4c-e182-467f-8d22-56389ad95d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aaf3e1-psg9p" Nov 01 23:19:17.025526 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:17.025497 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lsr22"] Nov 01 23:19:17.040303 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:17.040274 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-lsr22"] Nov 01 23:19:17.122892 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:17.122825 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks5rn\" (UniqueName: \"kubernetes.io/projected/79f34e4c-e182-467f-8d22-56389ad95d89-kube-api-access-ks5rn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-aaf3e1-psg9p\" (UID: \"79f34e4c-e182-467f-8d22-56389ad95d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aaf3e1-psg9p" Nov 01 23:19:17.123107 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:17.122915 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79f34e4c-e182-467f-8d22-56389ad95d89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-aaf3e1-psg9p\" (UID: \"79f34e4c-e182-467f-8d22-56389ad95d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aaf3e1-psg9p" Nov 01 23:19:17.123107 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:17.123004 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79f34e4c-e182-467f-8d22-56389ad95d89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-aaf3e1-psg9p\" (UID: \"79f34e4c-e182-467f-8d22-56389ad95d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aaf3e1-psg9p" Nov 01 23:19:17.123735 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:17.123703 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79f34e4c-e182-467f-8d22-56389ad95d89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-aaf3e1-psg9p\" (UID: \"79f34e4c-e182-467f-8d22-56389ad95d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aaf3e1-psg9p" Nov 01 23:19:17.125622 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:17.125596 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79f34e4c-e182-467f-8d22-56389ad95d89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-aaf3e1-psg9p\" (UID: \"79f34e4c-e182-467f-8d22-56389ad95d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aaf3e1-psg9p" Nov 01 23:19:17.130664 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:17.130641 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks5rn\" (UniqueName: \"kubernetes.io/projected/79f34e4c-e182-467f-8d22-56389ad95d89-kube-api-access-ks5rn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-aaf3e1-psg9p\" (UID: \"79f34e4c-e182-467f-8d22-56389ad95d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aaf3e1-psg9p" Nov 01 23:19:17.252339 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:17.252251 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aaf3e1-psg9p" Nov 01 23:19:17.541096 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:17.540902 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aaf3e1-psg9p"] Nov 01 23:19:17.544063 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:19:17.544025 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f34e4c_e182_467f_8d22_56389ad95d89.slice/crio-3e4cbc0fefd657f2c35f250c95a3fa93745349a4be1036e5db551778b74c0f79 WatchSource:0}: Error finding container 3e4cbc0fefd657f2c35f250c95a3fa93745349a4be1036e5db551778b74c0f79: Status 404 returned error can't find the container with id 3e4cbc0fefd657f2c35f250c95a3fa93745349a4be1036e5db551778b74c0f79 Nov 01 23:19:17.693282 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:17.693240 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:19:17.746038 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:17.746012 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccca4473-3959-4e16-af06-0133607bc80d" path="/var/lib/kubelet/pods/ccca4473-3959-4e16-af06-0133607bc80d/volumes" Nov 01 23:19:18.463061 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:18.463030 5011 generic.go:356] "Generic (PLEG): container finished" podID="79f34e4c-e182-467f-8d22-56389ad95d89" containerID="63a3b114b7854683f664674f9613585fe5cd22e1a9690f62e2ea59153f075da5" exitCode=0 Nov 01 23:19:18.463452 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:18.463112 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aaf3e1-psg9p" event={"ID":"79f34e4c-e182-467f-8d22-56389ad95d89","Type":"ContainerDied","Data":"63a3b114b7854683f664674f9613585fe5cd22e1a9690f62e2ea59153f075da5"} Nov 01 23:19:18.463452 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:18.463162 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aaf3e1-psg9p" event={"ID":"79f34e4c-e182-467f-8d22-56389ad95d89","Type":"ContainerStarted","Data":"3e4cbc0fefd657f2c35f250c95a3fa93745349a4be1036e5db551778b74c0f79"} Nov 01 23:19:19.602804 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:19.602780 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aaf3e1-psg9p" Nov 01 23:19:19.748509 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:19.748429 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79f34e4c-e182-467f-8d22-56389ad95d89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79f34e4c-e182-467f-8d22-56389ad95d89-maas-api-openshift-ca\") pod \"79f34e4c-e182-467f-8d22-56389ad95d89\" (UID: \"79f34e4c-e182-467f-8d22-56389ad95d89\") " Nov 01 23:19:19.748509 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:19.748468 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79f34e4c-e182-467f-8d22-56389ad95d89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79f34e4c-e182-467f-8d22-56389ad95d89-maas-api-xks-ca\") pod \"79f34e4c-e182-467f-8d22-56389ad95d89\" (UID: \"79f34e4c-e182-467f-8d22-56389ad95d89\") " Nov 01 23:19:19.748765 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:19.748606 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79f34e4c-e182-467f-8d22-56389ad95d89-kube-api-access-ks5rn\" (UniqueName: \"kubernetes.io/projected/79f34e4c-e182-467f-8d22-56389ad95d89-kube-api-access-ks5rn\") pod \"79f34e4c-e182-467f-8d22-56389ad95d89\" (UID: \"79f34e4c-e182-467f-8d22-56389ad95d89\") " Nov 01 23:19:19.748861 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:19.748805 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79f34e4c-e182-467f-8d22-56389ad95d89-maas-api-openshift-ca" pod "79f34e4c-e182-467f-8d22-56389ad95d89" (UID: "79f34e4c-e182-467f-8d22-56389ad95d89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:19:19.750634 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:19.750609 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79f34e4c-e182-467f-8d22-56389ad95d89-maas-api-xks-ca" pod "79f34e4c-e182-467f-8d22-56389ad95d89" (UID: "79f34e4c-e182-467f-8d22-56389ad95d89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:19:19.750772 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:19.750743 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f34e4c-e182-467f-8d22-56389ad95d89-kube-api-access-ks5rn" pod "79f34e4c-e182-467f-8d22-56389ad95d89" (UID: "79f34e4c-e182-467f-8d22-56389ad95d89"). InnerVolumeSpecName "kube-api-access-ks5rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:19:19.849869 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:19.849810 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks5rn\" (UniqueName: \"kubernetes.io/projected/79f34e4c-e182-467f-8d22-56389ad95d89-kube-api-access-ks5rn\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:19.849869 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:19.849862 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79f34e4c-e182-467f-8d22-56389ad95d89-maas-api-openshift-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:19.849869 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:19.849873 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79f34e4c-e182-467f-8d22-56389ad95d89-maas-api-xks-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:20.474797 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:20.474759 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aaf3e1-psg9p" Nov 01 23:19:20.474969 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:20.474792 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aaf3e1-psg9p" event={"ID":"79f34e4c-e182-467f-8d22-56389ad95d89","Type":"ContainerDied","Data":"3e4cbc0fefd657f2c35f250c95a3fa93745349a4be1036e5db551778b74c0f79"} Nov 01 23:19:20.474969 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:20.474891 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e4cbc0fefd657f2c35f250c95a3fa93745349a4be1036e5db551778b74c0f79" Nov 01 23:19:27.051674 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:27.051639 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-aaf3e1-5f577mnw6k"] Nov 01 23:19:27.860227 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:27.860188 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aaf3e1-psg9p"] Nov 01 23:19:27.861836 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:27.861803 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-aaf3e1-psg9p"] Nov 01 23:19:29.019628 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:29.019588 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:19:29.745531 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:29.745498 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79f34e4c-e182-467f-8d22-56389ad95d89" path="/var/lib/kubelet/pods/79f34e4c-e182-467f-8d22-56389ad95d89/volumes" Nov 01 23:19:32.052472 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:32.052348 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aaf3e1-5f577mnw6k" podUID="3633b776-74ea-4214-9f8d-627a695904cb" containerName="payload-pre-processing" containerID="cri-o://a3418dcf7f91cdcb453f0bebfcb92f4346fab9e229c143e1e30b2ca286f2f080" gracePeriod=25 Nov 01 23:19:33.151715 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:33.151677 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aaf3e1-5f577mnw6k" podUID="3633b776-74ea-4214-9f8d-627a695904cb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:19:37.019701 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:37.019664 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wjs4b"] Nov 01 23:19:37.023887 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:37.023859 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wjs4b"] Nov 01 23:19:37.747722 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:37.747688 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7" path="/var/lib/kubelet/pods/a11fe6f0-3ef5-4b91-8c9d-75af1561bbb7/volumes" Nov 01 23:19:43.151711 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:43.151673 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aaf3e1-5f577mnw6k" podUID="3633b776-74ea-4214-9f8d-627a695904cb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:19:46.959713 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:19:46.959676 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:19:47.282376 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:47.282294 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:19:51.513267 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:51.513230 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:19:52.303134 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:52.303096 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aaf3e1-5f577mnw6k" Nov 01 23:19:52.381201 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:52.381128 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3633b776-74ea-4214-9f8d-627a695904cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3633b776-74ea-4214-9f8d-627a695904cb-plugins-config-volume\") pod \"3633b776-74ea-4214-9f8d-627a695904cb\" (UID: \"3633b776-74ea-4214-9f8d-627a695904cb\") " Nov 01 23:19:52.381364 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:52.381234 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3633b776-74ea-4214-9f8d-627a695904cb-kube-api-access-6hsgr\" (UniqueName: \"kubernetes.io/projected/3633b776-74ea-4214-9f8d-627a695904cb-kube-api-access-6hsgr\") pod \"3633b776-74ea-4214-9f8d-627a695904cb\" (UID: \"3633b776-74ea-4214-9f8d-627a695904cb\") " Nov 01 23:19:52.381578 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:52.381554 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3633b776-74ea-4214-9f8d-627a695904cb-plugins-config-volume" pod "3633b776-74ea-4214-9f8d-627a695904cb" (UID: "3633b776-74ea-4214-9f8d-627a695904cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:19:52.383422 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:52.383404 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3633b776-74ea-4214-9f8d-627a695904cb-kube-api-access-6hsgr" pod "3633b776-74ea-4214-9f8d-627a695904cb" (UID: "3633b776-74ea-4214-9f8d-627a695904cb"). InnerVolumeSpecName "kube-api-access-6hsgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:19:52.482696 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:52.482657 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3633b776-74ea-4214-9f8d-627a695904cb-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:52.482696 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:52.482689 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hsgr\" (UniqueName: \"kubernetes.io/projected/3633b776-74ea-4214-9f8d-627a695904cb-kube-api-access-6hsgr\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:52.645066 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:52.644974 5011 generic.go:356] "Generic (PLEG): container finished" podID="3633b776-74ea-4214-9f8d-627a695904cb" containerID="a3418dcf7f91cdcb453f0bebfcb92f4346fab9e229c143e1e30b2ca286f2f080" exitCode=0 Nov 01 23:19:52.645066 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:52.645030 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aaf3e1-5f577mnw6k" Nov 01 23:19:52.645539 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:52.645062 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aaf3e1-5f577mnw6k" event={"ID":"3633b776-74ea-4214-9f8d-627a695904cb","Type":"ContainerDied","Data":"a3418dcf7f91cdcb453f0bebfcb92f4346fab9e229c143e1e30b2ca286f2f080"} Nov 01 23:19:52.645539 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:52.645120 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aaf3e1-5f577mnw6k" event={"ID":"3633b776-74ea-4214-9f8d-627a695904cb","Type":"ContainerDied","Data":"c39f9bbefd439437037661f798392ec0298c99daf04c3788a95d9bd7cb979ed3"} Nov 01 23:19:52.645539 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:52.645143 5011 scope.go:122] "RemoveContainer" containerID="a3418dcf7f91cdcb453f0bebfcb92f4346fab9e229c143e1e30b2ca286f2f080" Nov 01 23:19:52.654091 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:52.654076 5011 scope.go:122] "RemoveContainer" containerID="a3418dcf7f91cdcb453f0bebfcb92f4346fab9e229c143e1e30b2ca286f2f080" Nov 01 23:19:52.654353 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:19:52.654332 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3418dcf7f91cdcb453f0bebfcb92f4346fab9e229c143e1e30b2ca286f2f080\": container with ID starting with a3418dcf7f91cdcb453f0bebfcb92f4346fab9e229c143e1e30b2ca286f2f080 not found: ID does not exist" containerID="a3418dcf7f91cdcb453f0bebfcb92f4346fab9e229c143e1e30b2ca286f2f080" Nov 01 23:19:52.654431 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:52.654364 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3418dcf7f91cdcb453f0bebfcb92f4346fab9e229c143e1e30b2ca286f2f080"} err="failed to get container status \"a3418dcf7f91cdcb453f0bebfcb92f4346fab9e229c143e1e30b2ca286f2f080\": rpc error: code = NotFound desc = could not find container \"a3418dcf7f91cdcb453f0bebfcb92f4346fab9e229c143e1e30b2ca286f2f080\": container with ID starting with a3418dcf7f91cdcb453f0bebfcb92f4346fab9e229c143e1e30b2ca286f2f080 not found: ID does not exist" Nov 01 23:19:52.665475 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:52.665444 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-aaf3e1-5f577mnw6k"] Nov 01 23:19:52.668180 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:52.668160 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-aaf3e1-5f577mnw6k"] Nov 01 23:19:53.744664 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:53.744630 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3633b776-74ea-4214-9f8d-627a695904cb" path="/var/lib/kubelet/pods/3633b776-74ea-4214-9f8d-627a695904cb/volumes" Nov 01 23:19:55.739127 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:19:55.739097 5011 scope.go:122] "RemoveContainer" containerID="0b941e0282da1cc866b88d6de8ab71f2322fbcb83cd5a3768809a8c081d18458" Nov 01 23:20:15.021044 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:20:15.021003 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7sb5k"] Nov 01 23:20:15.023572 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:20:15.023272 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7sb5k"] Nov 01 23:20:15.745424 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:20:15.745388 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca" path="/var/lib/kubelet/pods/4c3ddb2d-fe16-42e1-95c8-d3b8d3e79eca/volumes" Nov 01 23:20:16.958910 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:20:16.958872 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:20:46.960502 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:20:46.960462 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:20:55.764432 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:20:55.764398 5011 scope.go:122] "RemoveContainer" containerID="c09cda1a7fde8b8fd194e306da345871ef2d5fb8fedfa2fe854c4bd1c915c987" Nov 01 23:20:55.772729 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:20:55.772708 5011 scope.go:122] "RemoveContainer" containerID="9ce07da4fb39d3def0352d09270b13764508641d23632a0796d5b02fa728aff1" Nov 01 23:21:13.509314 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:13.509234 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:21:16.958538 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:21:16.958492 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:21:28.504256 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:28.504219 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:21:32.010098 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:32.010057 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:21:46.201028 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:46.200990 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:21:46.959418 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:21:46.959381 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:21:51.894876 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:51.894845 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-45dd82-7bf64z2lpx"] Nov 01 23:21:51.895360 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:51.895259 5011 cpu_manager.go:414] "containerMap: removing container" podUID="3633b776-74ea-4214-9f8d-627a695904cb" containerName="payload-pre-processing" Nov 01 23:21:51.895360 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:51.895270 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3633b776-74ea-4214-9f8d-627a695904cb" containerName="payload-pre-processing" Nov 01 23:21:51.895360 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:51.895284 5011 cpu_manager.go:414] "containerMap: removing container" podUID="79f34e4c-e182-467f-8d22-56389ad95d89" containerName="revoke-keys" Nov 01 23:21:51.895360 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:51.895290 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79f34e4c-e182-467f-8d22-56389ad95d89" containerName="revoke-keys" Nov 01 23:21:51.895360 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:51.895349 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="3633b776-74ea-4214-9f8d-627a695904cb" containerName="payload-pre-processing" Nov 01 23:21:51.895360 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:51.895357 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="79f34e4c-e182-467f-8d22-56389ad95d89" containerName="revoke-keys" Nov 01 23:21:51.898504 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:51.898488 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-45dd82-7bf64z2lpx" Nov 01 23:21:51.906245 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:51.906218 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-fab9fbd4-dockercfg-zv2bt\"" Nov 01 23:21:51.906960 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:51.906937 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-45dd82\"" Nov 01 23:21:51.909561 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:51.909537 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-45dd82-7bf64z2lpx"] Nov 01 23:21:51.921645 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:51.921604 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-45dd82-86d5fd7dbgjkh5"] Nov 01 23:21:51.925772 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:51.925747 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-45dd82-86d5fd7dbgjkh5" Nov 01 23:21:51.937634 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:51.937600 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-45dd82-86d5fd7dbgjkh5"] Nov 01 23:21:51.951245 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:51.951215 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81c12a3a-aace-45de-a301-edb89299453d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-45dd82-7bf64z2lpx\" (UID: \"81c12a3a-aace-45de-a301-edb89299453d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-45dd82-7bf64z2lpx" Nov 01 23:21:51.951399 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:51.951288 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b031a9-8233-4f52-a414-66a39c2a13d9-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-45dd82-86d5fd7dbgjkh5\" (UID: \"85b031a9-8233-4f52-a414-66a39c2a13d9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-45dd82-86d5fd7dbgjkh5" Nov 01 23:21:51.951473 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:51.951451 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jlcq\" (UniqueName: \"kubernetes.io/projected/85b031a9-8233-4f52-a414-66a39c2a13d9-kube-api-access-9jlcq\") pod \"payload-processing-e2e-worker-single-main-45dd82-86d5fd7dbgjkh5\" (UID: \"85b031a9-8233-4f52-a414-66a39c2a13d9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-45dd82-86d5fd7dbgjkh5" Nov 01 23:21:51.951513 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:51.951489 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lcmj\" (UniqueName: \"kubernetes.io/projected/81c12a3a-aace-45de-a301-edb89299453d-kube-api-access-4lcmj\") pod \"payload-pre-processing-e2e-worker-single-main-45dd82-7bf64z2lpx\" (UID: \"81c12a3a-aace-45de-a301-edb89299453d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-45dd82-7bf64z2lpx" Nov 01 23:21:52.052552 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:52.052500 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jlcq\" (UniqueName: \"kubernetes.io/projected/85b031a9-8233-4f52-a414-66a39c2a13d9-kube-api-access-9jlcq\") pod \"payload-processing-e2e-worker-single-main-45dd82-86d5fd7dbgjkh5\" (UID: \"85b031a9-8233-4f52-a414-66a39c2a13d9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-45dd82-86d5fd7dbgjkh5" Nov 01 23:21:52.052552 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:52.052553 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lcmj\" (UniqueName: \"kubernetes.io/projected/81c12a3a-aace-45de-a301-edb89299453d-kube-api-access-4lcmj\") pod \"payload-pre-processing-e2e-worker-single-main-45dd82-7bf64z2lpx\" (UID: \"81c12a3a-aace-45de-a301-edb89299453d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-45dd82-7bf64z2lpx" Nov 01 23:21:52.052882 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:52.052587 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81c12a3a-aace-45de-a301-edb89299453d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-45dd82-7bf64z2lpx\" (UID: \"81c12a3a-aace-45de-a301-edb89299453d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-45dd82-7bf64z2lpx" Nov 01 23:21:52.052882 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:52.052639 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b031a9-8233-4f52-a414-66a39c2a13d9-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-45dd82-86d5fd7dbgjkh5\" (UID: \"85b031a9-8233-4f52-a414-66a39c2a13d9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-45dd82-86d5fd7dbgjkh5" Nov 01 23:21:52.053477 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:52.053451 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b031a9-8233-4f52-a414-66a39c2a13d9-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-45dd82-86d5fd7dbgjkh5\" (UID: \"85b031a9-8233-4f52-a414-66a39c2a13d9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-45dd82-86d5fd7dbgjkh5" Nov 01 23:21:52.053477 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:52.053463 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81c12a3a-aace-45de-a301-edb89299453d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-45dd82-7bf64z2lpx\" (UID: \"81c12a3a-aace-45de-a301-edb89299453d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-45dd82-7bf64z2lpx" Nov 01 23:21:52.061478 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:52.061456 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jlcq\" (UniqueName: \"kubernetes.io/projected/85b031a9-8233-4f52-a414-66a39c2a13d9-kube-api-access-9jlcq\") pod \"payload-processing-e2e-worker-single-main-45dd82-86d5fd7dbgjkh5\" (UID: \"85b031a9-8233-4f52-a414-66a39c2a13d9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-45dd82-86d5fd7dbgjkh5" Nov 01 23:21:52.061585 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:52.061542 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lcmj\" (UniqueName: \"kubernetes.io/projected/81c12a3a-aace-45de-a301-edb89299453d-kube-api-access-4lcmj\") pod \"payload-pre-processing-e2e-worker-single-main-45dd82-7bf64z2lpx\" (UID: \"81c12a3a-aace-45de-a301-edb89299453d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-45dd82-7bf64z2lpx" Nov 01 23:21:52.212132 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:52.212099 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-45dd82-7bf64z2lpx" Nov 01 23:21:52.242948 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:52.242920 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-45dd82-86d5fd7dbgjkh5" Nov 01 23:21:52.501824 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:52.501801 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-45dd82-7bf64z2lpx"] Nov 01 23:21:52.504653 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:21:52.504615 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81c12a3a_aace_45de_a301_edb89299453d.slice/crio-c2646607a5228f9108c481b0b1bfda8f2557d7ad4c39be84119c18df83ef7e1f WatchSource:0}: Error finding container c2646607a5228f9108c481b0b1bfda8f2557d7ad4c39be84119c18df83ef7e1f: Status 404 returned error can't find the container with id c2646607a5228f9108c481b0b1bfda8f2557d7ad4c39be84119c18df83ef7e1f Nov 01 23:21:52.585420 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:52.585393 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-45dd82-86d5fd7dbgjkh5"] Nov 01 23:21:52.587877 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:21:52.587819 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b031a9_8233_4f52_a414_66a39c2a13d9.slice/crio-3db1ae2cc8aaab167553839027a4db9efe3710fd93666ce3e208a271ad363465 WatchSource:0}: Error finding container 3db1ae2cc8aaab167553839027a4db9efe3710fd93666ce3e208a271ad363465: Status 404 returned error can't find the container with id 3db1ae2cc8aaab167553839027a4db9efe3710fd93666ce3e208a271ad363465 Nov 01 23:21:53.314146 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:53.314101 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-45dd82-7bf64z2lpx" event={"ID":"81c12a3a-aace-45de-a301-edb89299453d","Type":"ContainerStarted","Data":"7cb9190383559d6b4498f53d2b4c04d4be69b59d56eefe6f8018f0ec7dfc5cca"} Nov 01 23:21:53.314604 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:53.314162 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-45dd82-7bf64z2lpx" event={"ID":"81c12a3a-aace-45de-a301-edb89299453d","Type":"ContainerStarted","Data":"c2646607a5228f9108c481b0b1bfda8f2557d7ad4c39be84119c18df83ef7e1f"} Nov 01 23:21:53.316143 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:53.316111 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-45dd82-86d5fd7dbgjkh5" event={"ID":"85b031a9-8233-4f52-a414-66a39c2a13d9","Type":"ContainerStarted","Data":"6ad5e6eff56174e7c10baed9b34574b0fded8e05beb57ee50fa032353552ecd3"} Nov 01 23:21:53.316269 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:53.316170 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-45dd82-86d5fd7dbgjkh5" event={"ID":"85b031a9-8233-4f52-a414-66a39c2a13d9","Type":"ContainerStarted","Data":"3db1ae2cc8aaab167553839027a4db9efe3710fd93666ce3e208a271ad363465"} Nov 01 23:21:53.316269 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:53.316201 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-45dd82-86d5fd7dbgjkh5" Nov 01 23:21:53.352571 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:53.352516 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-45dd82-7bf64z2lpx" podStartSLOduration=2.352499951 podStartE2EDuration="2.352499951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:21:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:21:53.351254349 +0000 UTC m=+3481.755844983" watchObservedRunningTime="2026-11-01 23:21:53.352499951 +0000 UTC m=+3481.757090582" Nov 01 23:21:53.374908 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:53.374855 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-45dd82-86d5fd7dbgjkh5" podStartSLOduration=2.374823319 podStartE2EDuration="2.374823319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:21:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:21:53.372724201 +0000 UTC m=+3481.777314834" watchObservedRunningTime="2026-11-01 23:21:53.374823319 +0000 UTC m=+3481.779413948" Nov 01 23:21:54.018573 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:54.018535 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:21:54.321609 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:54.321518 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-45dd82-7bf64z2lpx" Nov 01 23:21:55.797586 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:21:55.797551 5011 scope.go:122] "RemoveContainer" containerID="6b7861f843cd30b340f9e21c4ec56477c09b1c7b02196467a49f89897cb4684e" Nov 01 23:22:04.332466 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:04.332066 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-45dd82-86d5fd7dbgjkh5" Nov 01 23:22:05.330458 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:05.330431 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-45dd82-7bf64z2lpx" Nov 01 23:22:08.428540 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:08.428502 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:22:16.959548 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:22:16.959511 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:22:25.795191 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:25.795144 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:22:37.488033 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:37.487957 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f2421d-jnffb"] Nov 01 23:22:37.491719 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:37.491703 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2421d-jnffb" Nov 01 23:22:37.498640 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:37.498604 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:22:37.498809 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:37.498791 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:22:37.499793 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:37.499769 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f2421d-jnffb"] Nov 01 23:22:37.553931 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:37.553898 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a19decc7-a11d-49de-b391-abd4ce7d6a54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33f2421d-jnffb\" (UID: \"a19decc7-a11d-49de-b391-abd4ce7d6a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2421d-jnffb" Nov 01 23:22:37.554143 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:37.554023 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a19decc7-a11d-49de-b391-abd4ce7d6a54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33f2421d-jnffb\" (UID: \"a19decc7-a11d-49de-b391-abd4ce7d6a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2421d-jnffb" Nov 01 23:22:37.554143 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:37.554060 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dkp2\" (UniqueName: \"kubernetes.io/projected/a19decc7-a11d-49de-b391-abd4ce7d6a54-kube-api-access-5dkp2\") pod \"maas-api-revoke-sub-33f2421d-jnffb\" (UID: \"a19decc7-a11d-49de-b391-abd4ce7d6a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2421d-jnffb" Nov 01 23:22:37.566859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:37.566806 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:22:37.566859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:37.566806 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:22:37.655420 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:37.655382 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a19decc7-a11d-49de-b391-abd4ce7d6a54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33f2421d-jnffb\" (UID: \"a19decc7-a11d-49de-b391-abd4ce7d6a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2421d-jnffb" Nov 01 23:22:37.655634 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:37.655462 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a19decc7-a11d-49de-b391-abd4ce7d6a54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33f2421d-jnffb\" (UID: \"a19decc7-a11d-49de-b391-abd4ce7d6a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2421d-jnffb" Nov 01 23:22:37.655634 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:37.655492 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dkp2\" (UniqueName: \"kubernetes.io/projected/a19decc7-a11d-49de-b391-abd4ce7d6a54-kube-api-access-5dkp2\") pod \"maas-api-revoke-sub-33f2421d-jnffb\" (UID: \"a19decc7-a11d-49de-b391-abd4ce7d6a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2421d-jnffb" Nov 01 23:22:37.656234 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:37.656212 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a19decc7-a11d-49de-b391-abd4ce7d6a54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33f2421d-jnffb\" (UID: \"a19decc7-a11d-49de-b391-abd4ce7d6a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2421d-jnffb" Nov 01 23:22:37.658031 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:37.658001 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a19decc7-a11d-49de-b391-abd4ce7d6a54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33f2421d-jnffb\" (UID: \"a19decc7-a11d-49de-b391-abd4ce7d6a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2421d-jnffb" Nov 01 23:22:37.664022 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:37.664000 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dkp2\" (UniqueName: \"kubernetes.io/projected/a19decc7-a11d-49de-b391-abd4ce7d6a54-kube-api-access-5dkp2\") pod \"maas-api-revoke-sub-33f2421d-jnffb\" (UID: \"a19decc7-a11d-49de-b391-abd4ce7d6a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2421d-jnffb" Nov 01 23:22:37.806600 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:37.806508 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2421d-jnffb" Nov 01 23:22:38.081698 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:38.081554 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f2421d-jnffb"] Nov 01 23:22:38.084151 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:22:38.084120 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda19decc7_a11d_49de_b391_abd4ce7d6a54.slice/crio-ddca143bf9d1d9fdcfd32f208b4aff6cc8753724d463e0cf2f5646fe1cd0a0f7 WatchSource:0}: Error finding container ddca143bf9d1d9fdcfd32f208b4aff6cc8753724d463e0cf2f5646fe1cd0a0f7: Status 404 returned error can't find the container with id ddca143bf9d1d9fdcfd32f208b4aff6cc8753724d463e0cf2f5646fe1cd0a0f7 Nov 01 23:22:38.564814 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:38.564773 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2421d-jnffb" event={"ID":"a19decc7-a11d-49de-b391-abd4ce7d6a54","Type":"ContainerStarted","Data":"e0c17f57109732eda4dfea0358c0c1ec4d59926f697b7e70bcc44cf00198fb38"} Nov 01 23:22:38.565230 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:38.564854 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2421d-jnffb" event={"ID":"a19decc7-a11d-49de-b391-abd4ce7d6a54","Type":"ContainerStarted","Data":"ddca143bf9d1d9fdcfd32f208b4aff6cc8753724d463e0cf2f5646fe1cd0a0f7"} Nov 01 23:22:38.611388 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:38.611280 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2421d-jnffb" podStartSLOduration=1.6112661209999999 podStartE2EDuration="1.611266121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:22:38.608567699 +0000 UTC m=+3527.013158330" watchObservedRunningTime="2026-11-01 23:22:38.611266121 +0000 UTC m=+3527.015856752" Nov 01 23:22:39.571497 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:39.571458 5011 generic.go:356] "Generic (PLEG): container finished" podID="a19decc7-a11d-49de-b391-abd4ce7d6a54" containerID="e0c17f57109732eda4dfea0358c0c1ec4d59926f697b7e70bcc44cf00198fb38" exitCode=0 Nov 01 23:22:39.572031 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:39.571549 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2421d-jnffb" event={"ID":"a19decc7-a11d-49de-b391-abd4ce7d6a54","Type":"ContainerDied","Data":"e0c17f57109732eda4dfea0358c0c1ec4d59926f697b7e70bcc44cf00198fb38"} Nov 01 23:22:40.722312 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:40.722283 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2421d-jnffb" Nov 01 23:22:40.786602 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:40.786566 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a19decc7-a11d-49de-b391-abd4ce7d6a54-kube-api-access-5dkp2\" (UniqueName: \"kubernetes.io/projected/a19decc7-a11d-49de-b391-abd4ce7d6a54-kube-api-access-5dkp2\") pod \"a19decc7-a11d-49de-b391-abd4ce7d6a54\" (UID: \"a19decc7-a11d-49de-b391-abd4ce7d6a54\") " Nov 01 23:22:40.786790 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:40.786689 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a19decc7-a11d-49de-b391-abd4ce7d6a54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a19decc7-a11d-49de-b391-abd4ce7d6a54-maas-api-openshift-ca\") pod \"a19decc7-a11d-49de-b391-abd4ce7d6a54\" (UID: \"a19decc7-a11d-49de-b391-abd4ce7d6a54\") " Nov 01 23:22:40.786790 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:40.786754 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a19decc7-a11d-49de-b391-abd4ce7d6a54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a19decc7-a11d-49de-b391-abd4ce7d6a54-maas-api-xks-ca\") pod \"a19decc7-a11d-49de-b391-abd4ce7d6a54\" (UID: \"a19decc7-a11d-49de-b391-abd4ce7d6a54\") " Nov 01 23:22:40.787088 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:40.787063 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a19decc7-a11d-49de-b391-abd4ce7d6a54-maas-api-openshift-ca" pod "a19decc7-a11d-49de-b391-abd4ce7d6a54" (UID: "a19decc7-a11d-49de-b391-abd4ce7d6a54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:22:40.787392 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:40.787369 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a19decc7-a11d-49de-b391-abd4ce7d6a54-maas-api-openshift-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:22:40.789035 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:40.789010 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a19decc7-a11d-49de-b391-abd4ce7d6a54-maas-api-xks-ca" pod "a19decc7-a11d-49de-b391-abd4ce7d6a54" (UID: "a19decc7-a11d-49de-b391-abd4ce7d6a54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:22:40.789222 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:40.789203 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a19decc7-a11d-49de-b391-abd4ce7d6a54-kube-api-access-5dkp2" pod "a19decc7-a11d-49de-b391-abd4ce7d6a54" (UID: "a19decc7-a11d-49de-b391-abd4ce7d6a54"). InnerVolumeSpecName "kube-api-access-5dkp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:22:40.887945 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:40.887826 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a19decc7-a11d-49de-b391-abd4ce7d6a54-maas-api-xks-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:22:40.887945 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:40.887892 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dkp2\" (UniqueName: \"kubernetes.io/projected/a19decc7-a11d-49de-b391-abd4ce7d6a54-kube-api-access-5dkp2\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:22:41.583182 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:41.583148 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2421d-jnffb" Nov 01 23:22:41.583342 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:41.583148 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2421d-jnffb" event={"ID":"a19decc7-a11d-49de-b391-abd4ce7d6a54","Type":"ContainerDied","Data":"ddca143bf9d1d9fdcfd32f208b4aff6cc8753724d463e0cf2f5646fe1cd0a0f7"} Nov 01 23:22:41.583342 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:41.583299 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddca143bf9d1d9fdcfd32f208b4aff6cc8753724d463e0cf2f5646fe1cd0a0f7" Nov 01 23:22:46.959513 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:22:46.959466 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:22:48.335676 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:48.335640 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-45dd82-dmttn"] Nov 01 23:22:48.336154 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:48.336137 5011 cpu_manager.go:414] "containerMap: removing container" podUID="a19decc7-a11d-49de-b391-abd4ce7d6a54" containerName="revoke-keys" Nov 01 23:22:48.336154 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:48.336155 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a19decc7-a11d-49de-b391-abd4ce7d6a54" containerName="revoke-keys" Nov 01 23:22:48.336255 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:48.336230 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="a19decc7-a11d-49de-b391-abd4ce7d6a54" containerName="revoke-keys" Nov 01 23:22:48.339063 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:48.339048 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-45dd82-dmttn" Nov 01 23:22:48.345252 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:48.345227 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:22:48.345435 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:48.345416 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:22:48.345523 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:48.345503 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:22:48.345784 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:48.345762 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:22:48.348744 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:48.348718 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-45dd82-dmttn"] Nov 01 23:22:48.473001 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:48.472962 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49801ee8-9be7-4e8e-9824-d945af1960e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-45dd82-dmttn\" (UID: \"49801ee8-9be7-4e8e-9824-d945af1960e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-45dd82-dmttn" Nov 01 23:22:48.473186 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:48.473014 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49801ee8-9be7-4e8e-9824-d945af1960e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-45dd82-dmttn\" (UID: \"49801ee8-9be7-4e8e-9824-d945af1960e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-45dd82-dmttn" Nov 01 23:22:48.473233 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:48.473208 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zms4k\" (UniqueName: \"kubernetes.io/projected/49801ee8-9be7-4e8e-9824-d945af1960e3-kube-api-access-zms4k\") pod \"maas-api-revoke-keys-e2e-worker-single-main-45dd82-dmttn\" (UID: \"49801ee8-9be7-4e8e-9824-d945af1960e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-45dd82-dmttn" Nov 01 23:22:48.498859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:48.498792 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:22:48.573962 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:48.573933 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zms4k\" (UniqueName: \"kubernetes.io/projected/49801ee8-9be7-4e8e-9824-d945af1960e3-kube-api-access-zms4k\") pod \"maas-api-revoke-keys-e2e-worker-single-main-45dd82-dmttn\" (UID: \"49801ee8-9be7-4e8e-9824-d945af1960e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-45dd82-dmttn" Nov 01 23:22:48.574165 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:48.573997 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49801ee8-9be7-4e8e-9824-d945af1960e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-45dd82-dmttn\" (UID: \"49801ee8-9be7-4e8e-9824-d945af1960e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-45dd82-dmttn" Nov 01 23:22:48.574165 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:48.574023 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49801ee8-9be7-4e8e-9824-d945af1960e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-45dd82-dmttn\" (UID: \"49801ee8-9be7-4e8e-9824-d945af1960e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-45dd82-dmttn" Nov 01 23:22:48.574736 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:48.574710 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49801ee8-9be7-4e8e-9824-d945af1960e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-45dd82-dmttn\" (UID: \"49801ee8-9be7-4e8e-9824-d945af1960e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-45dd82-dmttn" Nov 01 23:22:48.576566 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:48.576536 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49801ee8-9be7-4e8e-9824-d945af1960e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-45dd82-dmttn\" (UID: \"49801ee8-9be7-4e8e-9824-d945af1960e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-45dd82-dmttn" Nov 01 23:22:48.582197 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:48.582144 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zms4k\" (UniqueName: \"kubernetes.io/projected/49801ee8-9be7-4e8e-9824-d945af1960e3-kube-api-access-zms4k\") pod \"maas-api-revoke-keys-e2e-worker-single-main-45dd82-dmttn\" (UID: \"49801ee8-9be7-4e8e-9824-d945af1960e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-45dd82-dmttn" Nov 01 23:22:48.656855 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:48.656741 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-45dd82-dmttn" Nov 01 23:22:48.930372 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:48.930341 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-45dd82-dmttn"] Nov 01 23:22:48.932500 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:22:48.932477 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49801ee8_9be7_4e8e_9824_d945af1960e3.slice/crio-2e5180058b2a86efdbead2c66b3815aaa97101fddf57b75457ceabc4879690a8 WatchSource:0}: Error finding container 2e5180058b2a86efdbead2c66b3815aaa97101fddf57b75457ceabc4879690a8: Status 404 returned error can't find the container with id 2e5180058b2a86efdbead2c66b3815aaa97101fddf57b75457ceabc4879690a8 Nov 01 23:22:49.625955 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:49.625869 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-45dd82-dmttn" event={"ID":"49801ee8-9be7-4e8e-9824-d945af1960e3","Type":"ContainerStarted","Data":"2b79d09ea579505e19456747a8646a647fdfa0bfdb2f49936e403aa3bcdabd43"} Nov 01 23:22:49.625955 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:49.625930 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-45dd82-dmttn" event={"ID":"49801ee8-9be7-4e8e-9824-d945af1960e3","Type":"ContainerStarted","Data":"2e5180058b2a86efdbead2c66b3815aaa97101fddf57b75457ceabc4879690a8"} Nov 01 23:22:49.641943 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:49.641885 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-45dd82-dmttn" podStartSLOduration=1.641866467 podStartE2EDuration="1.641866467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:22:49.640585433 +0000 UTC m=+3538.045176067" watchObservedRunningTime="2026-11-01 23:22:49.641866467 +0000 UTC m=+3538.046457100" Nov 01 23:22:50.632237 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:50.632199 5011 generic.go:356] "Generic (PLEG): container finished" podID="49801ee8-9be7-4e8e-9824-d945af1960e3" containerID="2b79d09ea579505e19456747a8646a647fdfa0bfdb2f49936e403aa3bcdabd43" exitCode=0 Nov 01 23:22:50.632635 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:50.632285 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-45dd82-dmttn" event={"ID":"49801ee8-9be7-4e8e-9824-d945af1960e3","Type":"ContainerDied","Data":"2b79d09ea579505e19456747a8646a647fdfa0bfdb2f49936e403aa3bcdabd43"} Nov 01 23:22:51.774564 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:51.774536 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-45dd82-dmttn" Nov 01 23:22:51.924606 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:51.924581 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49801ee8-9be7-4e8e-9824-d945af1960e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49801ee8-9be7-4e8e-9824-d945af1960e3-maas-api-xks-ca\") pod \"49801ee8-9be7-4e8e-9824-d945af1960e3\" (UID: \"49801ee8-9be7-4e8e-9824-d945af1960e3\") " Nov 01 23:22:51.924753 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:51.924636 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49801ee8-9be7-4e8e-9824-d945af1960e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49801ee8-9be7-4e8e-9824-d945af1960e3-maas-api-openshift-ca\") pod \"49801ee8-9be7-4e8e-9824-d945af1960e3\" (UID: \"49801ee8-9be7-4e8e-9824-d945af1960e3\") " Nov 01 23:22:51.924753 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:51.924716 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49801ee8-9be7-4e8e-9824-d945af1960e3-kube-api-access-zms4k\" (UniqueName: \"kubernetes.io/projected/49801ee8-9be7-4e8e-9824-d945af1960e3-kube-api-access-zms4k\") pod \"49801ee8-9be7-4e8e-9824-d945af1960e3\" (UID: \"49801ee8-9be7-4e8e-9824-d945af1960e3\") " Nov 01 23:22:51.925105 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:51.925075 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49801ee8-9be7-4e8e-9824-d945af1960e3-maas-api-openshift-ca" pod "49801ee8-9be7-4e8e-9824-d945af1960e3" (UID: "49801ee8-9be7-4e8e-9824-d945af1960e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:22:51.925222 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:51.925194 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49801ee8-9be7-4e8e-9824-d945af1960e3-maas-api-openshift-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:22:51.927037 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:51.927014 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49801ee8-9be7-4e8e-9824-d945af1960e3-maas-api-xks-ca" pod "49801ee8-9be7-4e8e-9824-d945af1960e3" (UID: "49801ee8-9be7-4e8e-9824-d945af1960e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:22:51.927153 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:51.927031 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49801ee8-9be7-4e8e-9824-d945af1960e3-kube-api-access-zms4k" pod "49801ee8-9be7-4e8e-9824-d945af1960e3" (UID: "49801ee8-9be7-4e8e-9824-d945af1960e3"). InnerVolumeSpecName "kube-api-access-zms4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:22:52.026555 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:52.026521 5011 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49801ee8-9be7-4e8e-9824-d945af1960e3-maas-api-xks-ca\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:22:52.026555 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:52.026549 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zms4k\" (UniqueName: \"kubernetes.io/projected/49801ee8-9be7-4e8e-9824-d945af1960e3-kube-api-access-zms4k\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:22:52.642742 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:52.642710 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-45dd82-dmttn" Nov 01 23:22:52.642928 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:52.642719 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-45dd82-dmttn" event={"ID":"49801ee8-9be7-4e8e-9824-d945af1960e3","Type":"ContainerDied","Data":"2e5180058b2a86efdbead2c66b3815aaa97101fddf57b75457ceabc4879690a8"} Nov 01 23:22:52.642928 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:52.642872 5011 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e5180058b2a86efdbead2c66b3815aaa97101fddf57b75457ceabc4879690a8" Nov 01 23:22:58.479815 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:58.478245 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-45dd82-86d5fd7dbgjkh5"] Nov 01 23:22:58.488108 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:58.488049 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-45dd82-7bf64z2lpx"] Nov 01 23:22:59.857204 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:59.857168 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-45dd82-dmttn"] Nov 01 23:22:59.859858 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:22:59.859814 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-45dd82-dmttn"] Nov 01 23:23:01.252149 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:01.252115 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:23:01.745575 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:01.745534 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49801ee8-9be7-4e8e-9824-d945af1960e3" path="/var/lib/kubelet/pods/49801ee8-9be7-4e8e-9824-d945af1960e3/volumes" Nov 01 23:23:03.479146 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:03.479102 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-45dd82-86d5fd7dbgjkh5" podUID="85b031a9-8233-4f52-a414-66a39c2a13d9" containerName="payload-processing" containerID="cri-o://6ad5e6eff56174e7c10baed9b34574b0fded8e05beb57ee50fa032353552ecd3" gracePeriod=25 Nov 01 23:23:03.488633 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:03.488597 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-45dd82-7bf64z2lpx" podUID="81c12a3a-aace-45de-a301-edb89299453d" containerName="payload-pre-processing" containerID="cri-o://7cb9190383559d6b4498f53d2b4c04d4be69b59d56eefe6f8018f0ec7dfc5cca" gracePeriod=25 Nov 01 23:23:04.323077 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:04.323040 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-45dd82-86d5fd7dbgjkh5" podUID="85b031a9-8233-4f52-a414-66a39c2a13d9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:23:05.328463 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:05.328428 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-45dd82-7bf64z2lpx" podUID="81c12a3a-aace-45de-a301-edb89299453d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:23:12.607102 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:12.607067 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:23:14.325344 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:14.325292 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-45dd82-86d5fd7dbgjkh5" podUID="85b031a9-8233-4f52-a414-66a39c2a13d9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:23:15.350978 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:15.350934 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-45dd82-7bf64z2lpx" podUID="81c12a3a-aace-45de-a301-edb89299453d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:23:15.366865 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:15.364927 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-m4nvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:23:15.366865 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:15.364987 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-m4nvr" podUID="fd9832d7-c187-41b2-bf9d-de25b0ea455a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:23:16.958941 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:23:16.958907 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:23:23.731010 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:23.730982 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-45dd82-86d5fd7dbgjkh5" Nov 01 23:23:23.772589 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:23.772562 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-45dd82-7bf64z2lpx" Nov 01 23:23:23.789474 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:23.789440 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85b031a9-8233-4f52-a414-66a39c2a13d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b031a9-8233-4f52-a414-66a39c2a13d9-plugins-config-volume\") pod \"85b031a9-8233-4f52-a414-66a39c2a13d9\" (UID: \"85b031a9-8233-4f52-a414-66a39c2a13d9\") " Nov 01 23:23:23.789661 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:23.789583 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85b031a9-8233-4f52-a414-66a39c2a13d9-kube-api-access-9jlcq\" (UniqueName: \"kubernetes.io/projected/85b031a9-8233-4f52-a414-66a39c2a13d9-kube-api-access-9jlcq\") pod \"85b031a9-8233-4f52-a414-66a39c2a13d9\" (UID: \"85b031a9-8233-4f52-a414-66a39c2a13d9\") " Nov 01 23:23:23.790320 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:23.790291 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85b031a9-8233-4f52-a414-66a39c2a13d9-plugins-config-volume" pod "85b031a9-8233-4f52-a414-66a39c2a13d9" (UID: "85b031a9-8233-4f52-a414-66a39c2a13d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:23:23.792484 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:23.792461 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b031a9-8233-4f52-a414-66a39c2a13d9-kube-api-access-9jlcq" pod "85b031a9-8233-4f52-a414-66a39c2a13d9" (UID: "85b031a9-8233-4f52-a414-66a39c2a13d9"). InnerVolumeSpecName "kube-api-access-9jlcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:23:23.830330 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:23.830296 5011 generic.go:356] "Generic (PLEG): container finished" podID="81c12a3a-aace-45de-a301-edb89299453d" containerID="7cb9190383559d6b4498f53d2b4c04d4be69b59d56eefe6f8018f0ec7dfc5cca" exitCode=0 Nov 01 23:23:23.830521 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:23.830347 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-45dd82-7bf64z2lpx" Nov 01 23:23:23.830521 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:23.830381 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-45dd82-7bf64z2lpx" event={"ID":"81c12a3a-aace-45de-a301-edb89299453d","Type":"ContainerDied","Data":"7cb9190383559d6b4498f53d2b4c04d4be69b59d56eefe6f8018f0ec7dfc5cca"} Nov 01 23:23:23.830521 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:23.830442 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-45dd82-7bf64z2lpx" event={"ID":"81c12a3a-aace-45de-a301-edb89299453d","Type":"ContainerDied","Data":"c2646607a5228f9108c481b0b1bfda8f2557d7ad4c39be84119c18df83ef7e1f"} Nov 01 23:23:23.830521 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:23.830470 5011 scope.go:122] "RemoveContainer" containerID="7cb9190383559d6b4498f53d2b4c04d4be69b59d56eefe6f8018f0ec7dfc5cca" Nov 01 23:23:23.831865 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:23.831818 5011 generic.go:356] "Generic (PLEG): container finished" podID="85b031a9-8233-4f52-a414-66a39c2a13d9" containerID="6ad5e6eff56174e7c10baed9b34574b0fded8e05beb57ee50fa032353552ecd3" exitCode=0 Nov 01 23:23:23.831980 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:23.831889 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-45dd82-86d5fd7dbgjkh5" Nov 01 23:23:23.831980 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:23.831934 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-45dd82-86d5fd7dbgjkh5" event={"ID":"85b031a9-8233-4f52-a414-66a39c2a13d9","Type":"ContainerDied","Data":"6ad5e6eff56174e7c10baed9b34574b0fded8e05beb57ee50fa032353552ecd3"} Nov 01 23:23:23.831980 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:23.831977 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-45dd82-86d5fd7dbgjkh5" event={"ID":"85b031a9-8233-4f52-a414-66a39c2a13d9","Type":"ContainerDied","Data":"3db1ae2cc8aaab167553839027a4db9efe3710fd93666ce3e208a271ad363465"} Nov 01 23:23:23.843283 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:23.843267 5011 scope.go:122] "RemoveContainer" containerID="7cb9190383559d6b4498f53d2b4c04d4be69b59d56eefe6f8018f0ec7dfc5cca" Nov 01 23:23:23.843592 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:23:23.843567 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cb9190383559d6b4498f53d2b4c04d4be69b59d56eefe6f8018f0ec7dfc5cca\": container with ID starting with 7cb9190383559d6b4498f53d2b4c04d4be69b59d56eefe6f8018f0ec7dfc5cca not found: ID does not exist" containerID="7cb9190383559d6b4498f53d2b4c04d4be69b59d56eefe6f8018f0ec7dfc5cca" Nov 01 23:23:23.843699 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:23.843599 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cb9190383559d6b4498f53d2b4c04d4be69b59d56eefe6f8018f0ec7dfc5cca"} err="failed to get container status \"7cb9190383559d6b4498f53d2b4c04d4be69b59d56eefe6f8018f0ec7dfc5cca\": rpc error: code = NotFound desc = could not find container \"7cb9190383559d6b4498f53d2b4c04d4be69b59d56eefe6f8018f0ec7dfc5cca\": container with ID starting with 7cb9190383559d6b4498f53d2b4c04d4be69b59d56eefe6f8018f0ec7dfc5cca not found: ID does not exist" Nov 01 23:23:23.843699 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:23.843618 5011 scope.go:122] "RemoveContainer" containerID="6ad5e6eff56174e7c10baed9b34574b0fded8e05beb57ee50fa032353552ecd3" Nov 01 23:23:23.852196 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:23.852179 5011 scope.go:122] "RemoveContainer" containerID="6ad5e6eff56174e7c10baed9b34574b0fded8e05beb57ee50fa032353552ecd3" Nov 01 23:23:23.852447 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:23:23.852426 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ad5e6eff56174e7c10baed9b34574b0fded8e05beb57ee50fa032353552ecd3\": container with ID starting with 6ad5e6eff56174e7c10baed9b34574b0fded8e05beb57ee50fa032353552ecd3 not found: ID does not exist" containerID="6ad5e6eff56174e7c10baed9b34574b0fded8e05beb57ee50fa032353552ecd3" Nov 01 23:23:23.852497 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:23.852459 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ad5e6eff56174e7c10baed9b34574b0fded8e05beb57ee50fa032353552ecd3"} err="failed to get container status \"6ad5e6eff56174e7c10baed9b34574b0fded8e05beb57ee50fa032353552ecd3\": rpc error: code = NotFound desc = could not find container \"6ad5e6eff56174e7c10baed9b34574b0fded8e05beb57ee50fa032353552ecd3\": container with ID starting with 6ad5e6eff56174e7c10baed9b34574b0fded8e05beb57ee50fa032353552ecd3 not found: ID does not exist" Nov 01 23:23:23.857508 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:23.857487 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-45dd82-86d5fd7dbgjkh5"] Nov 01 23:23:23.862935 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:23.862915 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-45dd82-86d5fd7dbgjkh5"] Nov 01 23:23:23.890975 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:23.890952 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81c12a3a-aace-45de-a301-edb89299453d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81c12a3a-aace-45de-a301-edb89299453d-plugins-config-volume\") pod \"81c12a3a-aace-45de-a301-edb89299453d\" (UID: \"81c12a3a-aace-45de-a301-edb89299453d\") " Nov 01 23:23:23.891071 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:23.890997 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81c12a3a-aace-45de-a301-edb89299453d-kube-api-access-4lcmj\" (UniqueName: \"kubernetes.io/projected/81c12a3a-aace-45de-a301-edb89299453d-kube-api-access-4lcmj\") pod \"81c12a3a-aace-45de-a301-edb89299453d\" (UID: \"81c12a3a-aace-45de-a301-edb89299453d\") " Nov 01 23:23:23.891288 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:23.891269 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81c12a3a-aace-45de-a301-edb89299453d-plugins-config-volume" pod "81c12a3a-aace-45de-a301-edb89299453d" (UID: "81c12a3a-aace-45de-a301-edb89299453d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:23:23.891385 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:23.891369 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jlcq\" (UniqueName: \"kubernetes.io/projected/85b031a9-8233-4f52-a414-66a39c2a13d9-kube-api-access-9jlcq\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:23:23.891425 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:23.891392 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81c12a3a-aace-45de-a301-edb89299453d-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:23:23.891425 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:23.891404 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b031a9-8233-4f52-a414-66a39c2a13d9-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:23:23.893263 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:23.893248 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81c12a3a-aace-45de-a301-edb89299453d-kube-api-access-4lcmj" pod "81c12a3a-aace-45de-a301-edb89299453d" (UID: "81c12a3a-aace-45de-a301-edb89299453d"). InnerVolumeSpecName "kube-api-access-4lcmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:23:23.992318 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:23.992287 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lcmj\" (UniqueName: \"kubernetes.io/projected/81c12a3a-aace-45de-a301-edb89299453d-kube-api-access-4lcmj\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:23:24.152675 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:24.152641 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-45dd82-7bf64z2lpx"] Nov 01 23:23:24.155549 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:24.155524 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-45dd82-7bf64z2lpx"] Nov 01 23:23:25.745509 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:25.745475 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81c12a3a-aace-45de-a301-edb89299453d" path="/var/lib/kubelet/pods/81c12a3a-aace-45de-a301-edb89299453d/volumes" Nov 01 23:23:25.745924 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:25.745852 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85b031a9-8233-4f52-a414-66a39c2a13d9" path="/var/lib/kubelet/pods/85b031a9-8233-4f52-a414-66a39c2a13d9/volumes" Nov 01 23:23:29.206814 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:29.206778 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:23:32.425857 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:32.424957 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bxw6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:23:32.425857 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:32.425055 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" podUID="4688829c-8b38-4d0f-a56c-f20892644b09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:23:44.715609 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:44.715572 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:23:47.014474 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:23:47.014395 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:23:47.425056 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:47.425013 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bxw6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:23:47.425432 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:47.425397 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" podUID="4688829c-8b38-4d0f-a56c-f20892644b09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:23:56.514464 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:56.514421 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:23:59.708857 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:59.708275 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-z9x5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:23:59.708857 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:23:59.708346 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-z9x5s" podUID="734444e5-d903-47e2-9ed0-d7617fd3b2ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:24:02.017919 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:02.017118 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:24:02.443859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:02.440015 5011 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-bxw6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:24:02.443859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:02.440085 5011 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-bxw6l" podUID="4688829c-8b38-4d0f-a56c-f20892644b09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:24:09.021253 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:09.021202 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65ac655d-7r4mc"] Nov 01 23:24:09.023266 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:09.023237 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65ac655d-7r4mc"] Nov 01 23:24:09.745310 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:09.745265 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e573717-83e3-4002-a886-942b595c88a1" path="/var/lib/kubelet/pods/1e573717-83e3-4002-a886-942b595c88a1/volumes" Nov 01 23:24:11.297588 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:11.297551 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:24:15.540095 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:15.540055 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:24:16.959197 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:24:16.959159 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:24:31.502533 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:31.502499 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:24:39.842558 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.842526 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh"] Nov 01 23:24:39.842996 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.842910 5011 cpu_manager.go:414] "containerMap: removing container" podUID="85b031a9-8233-4f52-a414-66a39c2a13d9" containerName="payload-processing" Nov 01 23:24:39.842996 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.842921 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85b031a9-8233-4f52-a414-66a39c2a13d9" containerName="payload-processing" Nov 01 23:24:39.842996 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.842932 5011 cpu_manager.go:414] "containerMap: removing container" podUID="49801ee8-9be7-4e8e-9824-d945af1960e3" containerName="revoke-keys" Nov 01 23:24:39.842996 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.842938 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49801ee8-9be7-4e8e-9824-d945af1960e3" containerName="revoke-keys" Nov 01 23:24:39.842996 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.842959 5011 cpu_manager.go:414] "containerMap: removing container" podUID="81c12a3a-aace-45de-a301-edb89299453d" containerName="payload-pre-processing" Nov 01 23:24:39.842996 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.842964 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81c12a3a-aace-45de-a301-edb89299453d" containerName="payload-pre-processing" Nov 01 23:24:39.843253 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.843029 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="81c12a3a-aace-45de-a301-edb89299453d" containerName="payload-pre-processing" Nov 01 23:24:39.843253 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.843041 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="85b031a9-8233-4f52-a414-66a39c2a13d9" containerName="payload-processing" Nov 01 23:24:39.843253 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.843047 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="49801ee8-9be7-4e8e-9824-d945af1960e3" containerName="revoke-keys" Nov 01 23:24:39.846201 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.846185 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:39.857642 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.857609 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-417a41eb-openshift-default-dockercfg-2h5bn\"" Nov 01 23:24:39.862160 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.862133 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh"] Nov 01 23:24:39.885815 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.885782 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e29ed894-cc26-4a57-9faa-509ef78debcf-istio-token\") pod \"e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:39.885815 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.885818 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-istio-envoy\") pod \"e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:39.886116 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.885890 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e29ed894-cc26-4a57-9faa-509ef78debcf-istiod-ca-cert\") pod \"e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:39.886116 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.885925 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-workload-certs\") pod \"e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:39.886116 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.885951 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e29ed894-cc26-4a57-9faa-509ef78debcf-istio-podinfo\") pod \"e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:39.886116 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.885975 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qnfw\" (UniqueName: \"kubernetes.io/projected/e29ed894-cc26-4a57-9faa-509ef78debcf-kube-api-access-8qnfw\") pod \"e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:39.886116 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.885995 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-workload-socket\") pod \"e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:39.886116 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.886025 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-credential-socket\") pod \"e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:39.886116 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.886089 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-istio-data\") pod \"e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:39.988900 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.987272 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qnfw\" (UniqueName: \"kubernetes.io/projected/e29ed894-cc26-4a57-9faa-509ef78debcf-kube-api-access-8qnfw\") pod \"e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:39.988900 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.987393 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-workload-socket\") pod \"e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:39.988900 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.987444 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-credential-socket\") pod \"e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:39.988900 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.987486 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-istio-data\") pod \"e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:39.988900 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.987680 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e29ed894-cc26-4a57-9faa-509ef78debcf-istio-token\") pod \"e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:39.988900 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.987715 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-istio-envoy\") pod \"e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:39.988900 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.987746 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e29ed894-cc26-4a57-9faa-509ef78debcf-istiod-ca-cert\") pod \"e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:39.988900 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.987772 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-workload-certs\") pod \"e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:39.988900 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.987935 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e29ed894-cc26-4a57-9faa-509ef78debcf-istio-podinfo\") pod \"e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:39.988900 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.988369 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-istio-data\") pod \"e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:39.988900 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.988380 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-workload-socket\") pod \"e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:39.988900 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.988629 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-credential-socket\") pod \"e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:39.989581 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.989368 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e29ed894-cc26-4a57-9faa-509ef78debcf-istiod-ca-cert\") pod \"e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:39.989581 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.989376 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-workload-certs\") pod \"e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:39.997070 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:39.996614 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e29ed894-cc26-4a57-9faa-509ef78debcf-istio-podinfo\") pod \"e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:40.009890 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:40.002341 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-istio-envoy\") pod \"e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:40.009890 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:40.008472 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qnfw\" (UniqueName: \"kubernetes.io/projected/e29ed894-cc26-4a57-9faa-509ef78debcf-kube-api-access-8qnfw\") pod \"e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:40.012339 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:40.012307 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e29ed894-cc26-4a57-9faa-509ef78debcf-istio-token\") pod \"e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:40.161125 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:40.161022 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:40.455426 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:40.455398 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh"] Nov 01 23:24:40.457140 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:24:40.457111 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode29ed894_cc26_4a57_9faa_509ef78debcf.slice/crio-a7c9e0150470aeac7ef4d69448355c1ece6e2ea9977e78235fe854b82fee1105 WatchSource:0}: Error finding container a7c9e0150470aeac7ef4d69448355c1ece6e2ea9977e78235fe854b82fee1105: Status 404 returned error can't find the container with id a7c9e0150470aeac7ef4d69448355c1ece6e2ea9977e78235fe854b82fee1105 Nov 01 23:24:40.459293 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:40.459260 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:24:40.459380 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:40.459328 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:24:40.459380 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:40.459357 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:24:40.656547 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:40.656509 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7587df7975-bq245"] Nov 01 23:24:40.660889 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:40.660863 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7587df7975-bq245" Nov 01 23:24:40.672446 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:40.672413 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7587df7975-bq245"] Nov 01 23:24:40.673365 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:40.673343 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 23:24:40.674901 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:40.674180 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:24:40.674901 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:40.674284 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 01 23:24:40.675075 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:40.674350 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:24:40.677808 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:40.677041 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-57phm\"" Nov 01 23:24:40.696012 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:40.695826 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc13d9fa-de9b-4846-b294-7ba3275093ba-maas-api-tls\") pod \"maas-api-7587df7975-bq245\" (UID: \"cc13d9fa-de9b-4846-b294-7ba3275093ba\") " pod="odh-ai-gateway-infra/maas-api-7587df7975-bq245" Nov 01 23:24:40.696012 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:40.695904 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc13d9fa-de9b-4846-b294-7ba3275093ba-openshift-service-ca\") pod \"maas-api-7587df7975-bq245\" (UID: \"cc13d9fa-de9b-4846-b294-7ba3275093ba\") " pod="odh-ai-gateway-infra/maas-api-7587df7975-bq245" Nov 01 23:24:40.696012 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:40.695989 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlkws\" (UniqueName: \"kubernetes.io/projected/cc13d9fa-de9b-4846-b294-7ba3275093ba-kube-api-access-dlkws\") pod \"maas-api-7587df7975-bq245\" (UID: \"cc13d9fa-de9b-4846-b294-7ba3275093ba\") " pod="odh-ai-gateway-infra/maas-api-7587df7975-bq245" Nov 01 23:24:40.696318 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:40.696095 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc13d9fa-de9b-4846-b294-7ba3275093ba-metrics-certs\") pod \"maas-api-7587df7975-bq245\" (UID: \"cc13d9fa-de9b-4846-b294-7ba3275093ba\") " pod="odh-ai-gateway-infra/maas-api-7587df7975-bq245" Nov 01 23:24:40.796886 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:40.796761 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc13d9fa-de9b-4846-b294-7ba3275093ba-openshift-service-ca\") pod \"maas-api-7587df7975-bq245\" (UID: \"cc13d9fa-de9b-4846-b294-7ba3275093ba\") " pod="odh-ai-gateway-infra/maas-api-7587df7975-bq245" Nov 01 23:24:40.796886 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:40.796817 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlkws\" (UniqueName: \"kubernetes.io/projected/cc13d9fa-de9b-4846-b294-7ba3275093ba-kube-api-access-dlkws\") pod \"maas-api-7587df7975-bq245\" (UID: \"cc13d9fa-de9b-4846-b294-7ba3275093ba\") " pod="odh-ai-gateway-infra/maas-api-7587df7975-bq245" Nov 01 23:24:40.796886 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:40.796870 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc13d9fa-de9b-4846-b294-7ba3275093ba-metrics-certs\") pod \"maas-api-7587df7975-bq245\" (UID: \"cc13d9fa-de9b-4846-b294-7ba3275093ba\") " pod="odh-ai-gateway-infra/maas-api-7587df7975-bq245" Nov 01 23:24:40.797221 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:40.796986 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc13d9fa-de9b-4846-b294-7ba3275093ba-maas-api-tls\") pod \"maas-api-7587df7975-bq245\" (UID: \"cc13d9fa-de9b-4846-b294-7ba3275093ba\") " pod="odh-ai-gateway-infra/maas-api-7587df7975-bq245" Nov 01 23:24:40.797519 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:40.797490 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc13d9fa-de9b-4846-b294-7ba3275093ba-openshift-service-ca\") pod \"maas-api-7587df7975-bq245\" (UID: \"cc13d9fa-de9b-4846-b294-7ba3275093ba\") " pod="odh-ai-gateway-infra/maas-api-7587df7975-bq245" Nov 01 23:24:40.799616 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:40.799593 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc13d9fa-de9b-4846-b294-7ba3275093ba-metrics-certs\") pod \"maas-api-7587df7975-bq245\" (UID: \"cc13d9fa-de9b-4846-b294-7ba3275093ba\") " pod="odh-ai-gateway-infra/maas-api-7587df7975-bq245" Nov 01 23:24:40.799720 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:40.799676 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc13d9fa-de9b-4846-b294-7ba3275093ba-maas-api-tls\") pod \"maas-api-7587df7975-bq245\" (UID: \"cc13d9fa-de9b-4846-b294-7ba3275093ba\") " pod="odh-ai-gateway-infra/maas-api-7587df7975-bq245" Nov 01 23:24:40.805515 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:40.805492 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlkws\" (UniqueName: \"kubernetes.io/projected/cc13d9fa-de9b-4846-b294-7ba3275093ba-kube-api-access-dlkws\") pod \"maas-api-7587df7975-bq245\" (UID: \"cc13d9fa-de9b-4846-b294-7ba3275093ba\") " pod="odh-ai-gateway-infra/maas-api-7587df7975-bq245" Nov 01 23:24:40.995368 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:40.995328 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7587df7975-bq245" Nov 01 23:24:41.272744 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:41.272663 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7587df7975-bq245"] Nov 01 23:24:41.275158 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:24:41.275130 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc13d9fa_de9b_4846_b294_7ba3275093ba.slice/crio-ee8306ca6a2ad104103e20a24b54df77bfef1212f43927fff3e04c54bf7c9e3d WatchSource:0}: Error finding container ee8306ca6a2ad104103e20a24b54df77bfef1212f43927fff3e04c54bf7c9e3d: Status 404 returned error can't find the container with id ee8306ca6a2ad104103e20a24b54df77bfef1212f43927fff3e04c54bf7c9e3d Nov 01 23:24:41.276569 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:41.276547 5011 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 23:24:41.358425 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:41.358337 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7587df7975-bq245" event={"ID":"cc13d9fa-de9b-4846-b294-7ba3275093ba","Type":"ContainerStarted","Data":"ee8306ca6a2ad104103e20a24b54df77bfef1212f43927fff3e04c54bf7c9e3d"} Nov 01 23:24:41.359940 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:41.359907 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" event={"ID":"e29ed894-cc26-4a57-9faa-509ef78debcf","Type":"ContainerStarted","Data":"2bba6826050bb13c14d628085f8f152ddb4e8c6a13aaa3a950eae60f0ffadc31"} Nov 01 23:24:41.360083 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:41.359955 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" event={"ID":"e29ed894-cc26-4a57-9faa-509ef78debcf","Type":"ContainerStarted","Data":"a7c9e0150470aeac7ef4d69448355c1ece6e2ea9977e78235fe854b82fee1105"} Nov 01 23:24:41.380049 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:41.379997 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" podStartSLOduration=2.379983063 podStartE2EDuration="2.379983063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:24:41.377537797 +0000 UTC m=+3649.782128429" watchObservedRunningTime="2026-11-01 23:24:41.379983063 +0000 UTC m=+3649.784573692" Nov 01 23:24:42.162270 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:42.162178 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:42.168284 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:42.168254 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:42.366803 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:42.366759 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7587df7975-bq245" event={"ID":"cc13d9fa-de9b-4846-b294-7ba3275093ba","Type":"ContainerStarted","Data":"7cdb6b5d57582dae767507c1420f60cff799479af61ea2e3f7f49d8abb51907c"} Nov 01 23:24:42.367040 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:42.367010 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:42.367197 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:42.367067 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7587df7975-bq245" Nov 01 23:24:42.367937 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:42.367919 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:42.384992 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:42.384946 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7587df7975-bq245" podStartSLOduration=1.868768545 podStartE2EDuration="2.384931569s" totalImagesPullingTime="516.163024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:24:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:24:42.384017147 +0000 UTC m=+3650.788607774" watchObservedRunningTime="2026-11-01 23:24:42.384931569 +0000 UTC m=+3650.789522201" Nov 01 23:24:46.412782 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:46.412744 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:24:46.963526 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:24:46.963492 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:24:48.377491 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:48.377462 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7587df7975-bq245" Nov 01 23:24:56.450625 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:56.450592 5011 generic.go:356] "Generic (PLEG): container finished" podID="e29ed894-cc26-4a57-9faa-509ef78debcf" containerID="2bba6826050bb13c14d628085f8f152ddb4e8c6a13aaa3a950eae60f0ffadc31" exitCode=0 Nov 01 23:24:56.451082 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:56.450661 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" event={"ID":"e29ed894-cc26-4a57-9faa-509ef78debcf","Type":"ContainerDied","Data":"2bba6826050bb13c14d628085f8f152ddb4e8c6a13aaa3a950eae60f0ffadc31"} Nov 01 23:24:56.451082 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:56.451063 5011 scope.go:122] "RemoveContainer" containerID="2bba6826050bb13c14d628085f8f152ddb4e8c6a13aaa3a950eae60f0ffadc31" Nov 01 23:24:56.452508 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:56.452473 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:24:56.452647 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:56.452561 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:24:56.452647 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:56.452598 5011 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 23:24:57.367129 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:57.367082 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:57.472416 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:57.472380 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" event={"ID":"e29ed894-cc26-4a57-9faa-509ef78debcf","Type":"ContainerStarted","Data":"bdd2f03047e17ffb5dc32640ab96ffea67bb6e001edc44e564074f72df26affb"} Nov 01 23:24:58.166079 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:58.166037 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:24:58.167363 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:58.167339 5011 patch_prober.go:29] interesting pod/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Nov 01 23:24:58.167483 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:58.167408 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" podUID="e29ed894-cc26-4a57-9faa-509ef78debcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Nov 01 23:24:59.166987 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:59.166953 5011 patch_prober.go:29] interesting pod/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Nov 01 23:24:59.167419 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:24:59.167008 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" podUID="e29ed894-cc26-4a57-9faa-509ef78debcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Nov 01 23:25:00.162856 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:00.162255 5011 patch_prober.go:29] interesting pod/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Nov 01 23:25:00.162856 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:00.162319 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" podUID="e29ed894-cc26-4a57-9faa-509ef78debcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Nov 01 23:25:01.165859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:01.165279 5011 patch_prober.go:29] interesting pod/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Nov 01 23:25:01.165859 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:01.165360 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" podUID="e29ed894-cc26-4a57-9faa-509ef78debcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Nov 01 23:25:02.166291 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:02.166254 5011 patch_prober.go:29] interesting pod/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body= Nov 01 23:25:02.166723 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:02.166315 5011 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" podUID="e29ed894-cc26-4a57-9faa-509ef78debcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" Nov 01 23:25:03.165937 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:03.165909 5011 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:25:03.166224 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:03.166201 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:25:03.166997 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:03.166978 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:25:06.101841 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:06.101797 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:25:09.745183 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:09.745148 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh"] Nov 01 23:25:09.745696 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:09.745358 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" podUID="e29ed894-cc26-4a57-9faa-509ef78debcf" containerName="istio-proxy" containerID="cri-o://bdd2f03047e17ffb5dc32640ab96ffea67bb6e001edc44e564074f72df26affb" gracePeriod=30 Nov 01 23:25:15.016188 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.016127 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:25:15.181661 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.181621 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e29ed894-cc26-4a57-9faa-509ef78debcf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e29ed894-cc26-4a57-9faa-509ef78debcf-istio-podinfo\") pod \"e29ed894-cc26-4a57-9faa-509ef78debcf\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " Nov 01 23:25:15.181874 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.181681 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-workload-socket\") pod \"e29ed894-cc26-4a57-9faa-509ef78debcf\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " Nov 01 23:25:15.181874 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.181714 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e29ed894-cc26-4a57-9faa-509ef78debcf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e29ed894-cc26-4a57-9faa-509ef78debcf-istiod-ca-cert\") pod \"e29ed894-cc26-4a57-9faa-509ef78debcf\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " Nov 01 23:25:15.181874 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.181813 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-credential-socket\") pod \"e29ed894-cc26-4a57-9faa-509ef78debcf\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " Nov 01 23:25:15.182043 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.181884 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e29ed894-cc26-4a57-9faa-509ef78debcf-kube-api-access-8qnfw\" (UniqueName: \"kubernetes.io/projected/e29ed894-cc26-4a57-9faa-509ef78debcf-kube-api-access-8qnfw\") pod \"e29ed894-cc26-4a57-9faa-509ef78debcf\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " Nov 01 23:25:15.182043 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.181916 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-istio-envoy\") pod \"e29ed894-cc26-4a57-9faa-509ef78debcf\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " Nov 01 23:25:15.182043 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.181964 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e29ed894-cc26-4a57-9faa-509ef78debcf-istio-token\" (UniqueName: \"kubernetes.io/projected/e29ed894-cc26-4a57-9faa-509ef78debcf-istio-token\") pod \"e29ed894-cc26-4a57-9faa-509ef78debcf\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " Nov 01 23:25:15.182043 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.181998 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-istio-data\") pod \"e29ed894-cc26-4a57-9faa-509ef78debcf\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " Nov 01 23:25:15.182270 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.182044 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-workload-certs\") pod \"e29ed894-cc26-4a57-9faa-509ef78debcf\" (UID: \"e29ed894-cc26-4a57-9faa-509ef78debcf\") " Nov 01 23:25:15.182270 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.182067 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-workload-socket" pod "e29ed894-cc26-4a57-9faa-509ef78debcf" (UID: "e29ed894-cc26-4a57-9faa-509ef78debcf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:25:15.182270 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.182177 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-credential-socket" pod "e29ed894-cc26-4a57-9faa-509ef78debcf" (UID: "e29ed894-cc26-4a57-9faa-509ef78debcf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:25:15.182270 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.182184 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e29ed894-cc26-4a57-9faa-509ef78debcf-istiod-ca-cert" pod "e29ed894-cc26-4a57-9faa-509ef78debcf" (UID: "e29ed894-cc26-4a57-9faa-509ef78debcf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:25:15.182494 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.182308 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-workload-certs" pod "e29ed894-cc26-4a57-9faa-509ef78debcf" (UID: "e29ed894-cc26-4a57-9faa-509ef78debcf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:25:15.182494 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.182450 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-istio-data" pod "e29ed894-cc26-4a57-9faa-509ef78debcf" (UID: "e29ed894-cc26-4a57-9faa-509ef78debcf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:25:15.182759 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.182736 5011 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-workload-certs\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:25:15.182759 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.182761 5011 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-workload-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:25:15.182952 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.182774 5011 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e29ed894-cc26-4a57-9faa-509ef78debcf-istiod-ca-cert\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:25:15.182952 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.182786 5011 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-credential-socket\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:25:15.182952 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.182797 5011 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-istio-data\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:25:15.184306 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.184274 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-istio-envoy" pod "e29ed894-cc26-4a57-9faa-509ef78debcf" (UID: "e29ed894-cc26-4a57-9faa-509ef78debcf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:25:15.184399 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.184335 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e29ed894-cc26-4a57-9faa-509ef78debcf-istio-token" pod "e29ed894-cc26-4a57-9faa-509ef78debcf" (UID: "e29ed894-cc26-4a57-9faa-509ef78debcf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:25:15.184664 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.184640 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e29ed894-cc26-4a57-9faa-509ef78debcf-kube-api-access-8qnfw" pod "e29ed894-cc26-4a57-9faa-509ef78debcf" (UID: "e29ed894-cc26-4a57-9faa-509ef78debcf"). InnerVolumeSpecName "kube-api-access-8qnfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:25:15.184731 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.184642 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e29ed894-cc26-4a57-9faa-509ef78debcf-istio-podinfo" pod "e29ed894-cc26-4a57-9faa-509ef78debcf" (UID: "e29ed894-cc26-4a57-9faa-509ef78debcf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:25:15.283408 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.283318 5011 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e29ed894-cc26-4a57-9faa-509ef78debcf-istio-token\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:25:15.283408 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.283349 5011 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e29ed894-cc26-4a57-9faa-509ef78debcf-istio-podinfo\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:25:15.283408 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.283359 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qnfw\" (UniqueName: \"kubernetes.io/projected/e29ed894-cc26-4a57-9faa-509ef78debcf-kube-api-access-8qnfw\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:25:15.283408 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.283369 5011 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e29ed894-cc26-4a57-9faa-509ef78debcf-istio-envoy\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:25:15.561731 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.561641 5011 generic.go:356] "Generic (PLEG): container finished" podID="e29ed894-cc26-4a57-9faa-509ef78debcf" containerID="bdd2f03047e17ffb5dc32640ab96ffea67bb6e001edc44e564074f72df26affb" exitCode=0 Nov 01 23:25:15.561731 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.561692 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" Nov 01 23:25:15.562003 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.561724 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" event={"ID":"e29ed894-cc26-4a57-9faa-509ef78debcf","Type":"ContainerDied","Data":"bdd2f03047e17ffb5dc32640ab96ffea67bb6e001edc44e564074f72df26affb"} Nov 01 23:25:15.562003 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.561779 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh" event={"ID":"e29ed894-cc26-4a57-9faa-509ef78debcf","Type":"ContainerDied","Data":"a7c9e0150470aeac7ef4d69448355c1ece6e2ea9977e78235fe854b82fee1105"} Nov 01 23:25:15.562003 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.561802 5011 scope.go:122] "RemoveContainer" containerID="bdd2f03047e17ffb5dc32640ab96ffea67bb6e001edc44e564074f72df26affb" Nov 01 23:25:15.574005 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.573985 5011 scope.go:122] "RemoveContainer" containerID="2bba6826050bb13c14d628085f8f152ddb4e8c6a13aaa3a950eae60f0ffadc31" Nov 01 23:25:15.583005 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.582978 5011 scope.go:122] "RemoveContainer" containerID="bdd2f03047e17ffb5dc32640ab96ffea67bb6e001edc44e564074f72df26affb" Nov 01 23:25:15.583289 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:25:15.583270 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdd2f03047e17ffb5dc32640ab96ffea67bb6e001edc44e564074f72df26affb\": container with ID starting with bdd2f03047e17ffb5dc32640ab96ffea67bb6e001edc44e564074f72df26affb not found: ID does not exist" containerID="bdd2f03047e17ffb5dc32640ab96ffea67bb6e001edc44e564074f72df26affb" Nov 01 23:25:15.583346 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.583300 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdd2f03047e17ffb5dc32640ab96ffea67bb6e001edc44e564074f72df26affb"} err="failed to get container status \"bdd2f03047e17ffb5dc32640ab96ffea67bb6e001edc44e564074f72df26affb\": rpc error: code = NotFound desc = could not find container \"bdd2f03047e17ffb5dc32640ab96ffea67bb6e001edc44e564074f72df26affb\": container with ID starting with bdd2f03047e17ffb5dc32640ab96ffea67bb6e001edc44e564074f72df26affb not found: ID does not exist" Nov 01 23:25:15.583346 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.583321 5011 scope.go:122] "RemoveContainer" containerID="2bba6826050bb13c14d628085f8f152ddb4e8c6a13aaa3a950eae60f0ffadc31" Nov 01 23:25:15.583599 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:25:15.583578 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bba6826050bb13c14d628085f8f152ddb4e8c6a13aaa3a950eae60f0ffadc31\": container with ID starting with 2bba6826050bb13c14d628085f8f152ddb4e8c6a13aaa3a950eae60f0ffadc31 not found: ID does not exist" containerID="2bba6826050bb13c14d628085f8f152ddb4e8c6a13aaa3a950eae60f0ffadc31" Nov 01 23:25:15.583670 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.583610 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bba6826050bb13c14d628085f8f152ddb4e8c6a13aaa3a950eae60f0ffadc31"} err="failed to get container status \"2bba6826050bb13c14d628085f8f152ddb4e8c6a13aaa3a950eae60f0ffadc31\": rpc error: code = NotFound desc = could not find container \"2bba6826050bb13c14d628085f8f152ddb4e8c6a13aaa3a950eae60f0ffadc31\": container with ID starting with 2bba6826050bb13c14d628085f8f152ddb4e8c6a13aaa3a950eae60f0ffadc31 not found: ID does not exist" Nov 01 23:25:15.587994 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.587970 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh"] Nov 01 23:25:15.592115 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.592092 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-417a41eb-openshift-default-bd6bd98f5-4xsbh"] Nov 01 23:25:15.746252 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:15.746217 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e29ed894-cc26-4a57-9faa-509ef78debcf" path="/var/lib/kubelet/pods/e29ed894-cc26-4a57-9faa-509ef78debcf/volumes" Nov 01 23:25:16.196489 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:16.196456 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-ede84d-79fd9b8df5-4r8vk"] Nov 01 23:25:16.197078 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:16.197057 5011 cpu_manager.go:414] "containerMap: removing container" podUID="e29ed894-cc26-4a57-9faa-509ef78debcf" containerName="istio-proxy" Nov 01 23:25:16.197182 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:16.197080 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e29ed894-cc26-4a57-9faa-509ef78debcf" containerName="istio-proxy" Nov 01 23:25:16.197241 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:16.197187 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="e29ed894-cc26-4a57-9faa-509ef78debcf" containerName="istio-proxy" Nov 01 23:25:16.201846 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:16.201805 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ede84d-79fd9b8df5-4r8vk" Nov 01 23:25:16.210207 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:16.210180 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-ede84d-79fd9b8df5-4r8vk"] Nov 01 23:25:16.210518 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:16.210490 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-ede84d\"" Nov 01 23:25:16.210647 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:16.210521 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-ede84d-dockercfg-cnz8w\"" Nov 01 23:25:16.242237 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:16.242208 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-ede84d-6c6c4ddd87-722rr"] Nov 01 23:25:16.242616 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:16.242604 5011 cpu_manager.go:414] "containerMap: removing container" podUID="e29ed894-cc26-4a57-9faa-509ef78debcf" containerName="istio-proxy" Nov 01 23:25:16.242669 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:16.242618 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e29ed894-cc26-4a57-9faa-509ef78debcf" containerName="istio-proxy" Nov 01 23:25:16.242717 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:16.242689 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="e29ed894-cc26-4a57-9faa-509ef78debcf" containerName="istio-proxy" Nov 01 23:25:16.245731 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:16.245715 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ede84d-6c6c4ddd87-722rr" Nov 01 23:25:16.256100 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:16.256078 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-ede84d-6c6c4ddd87-722rr"] Nov 01 23:25:16.307885 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:16.299385 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk6bv\" (UniqueName: \"kubernetes.io/projected/c2fabbd7-9e87-4417-9a52-544dc74cba58-kube-api-access-dk6bv\") pod \"payload-pre-processing-e2e-authcfg-http-ede84d-79fd9b8df5-4r8vk\" (UID: \"c2fabbd7-9e87-4417-9a52-544dc74cba58\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ede84d-79fd9b8df5-4r8vk" Nov 01 23:25:16.307885 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:16.299511 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2fabbd7-9e87-4417-9a52-544dc74cba58-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-ede84d-79fd9b8df5-4r8vk\" (UID: \"c2fabbd7-9e87-4417-9a52-544dc74cba58\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ede84d-79fd9b8df5-4r8vk" Nov 01 23:25:16.400257 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:16.400217 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk6bv\" (UniqueName: \"kubernetes.io/projected/c2fabbd7-9e87-4417-9a52-544dc74cba58-kube-api-access-dk6bv\") pod \"payload-pre-processing-e2e-authcfg-http-ede84d-79fd9b8df5-4r8vk\" (UID: \"c2fabbd7-9e87-4417-9a52-544dc74cba58\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ede84d-79fd9b8df5-4r8vk" Nov 01 23:25:16.400461 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:16.400278 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3123603-33bb-4484-a1b4-dccc87479653-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-ede84d-6c6c4ddd87-722rr\" (UID: \"b3123603-33bb-4484-a1b4-dccc87479653\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-ede84d-6c6c4ddd87-722rr" Nov 01 23:25:16.400461 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:16.400348 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt848\" (UniqueName: \"kubernetes.io/projected/b3123603-33bb-4484-a1b4-dccc87479653-kube-api-access-gt848\") pod \"payload-processing-e2e-authcfg-http-ede84d-6c6c4ddd87-722rr\" (UID: \"b3123603-33bb-4484-a1b4-dccc87479653\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-ede84d-6c6c4ddd87-722rr" Nov 01 23:25:16.400461 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:16.400433 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2fabbd7-9e87-4417-9a52-544dc74cba58-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-ede84d-79fd9b8df5-4r8vk\" (UID: \"c2fabbd7-9e87-4417-9a52-544dc74cba58\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ede84d-79fd9b8df5-4r8vk" Nov 01 23:25:16.401323 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:16.401295 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2fabbd7-9e87-4417-9a52-544dc74cba58-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-ede84d-79fd9b8df5-4r8vk\" (UID: \"c2fabbd7-9e87-4417-9a52-544dc74cba58\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ede84d-79fd9b8df5-4r8vk" Nov 01 23:25:16.409046 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:16.409022 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk6bv\" (UniqueName: \"kubernetes.io/projected/c2fabbd7-9e87-4417-9a52-544dc74cba58-kube-api-access-dk6bv\") pod \"payload-pre-processing-e2e-authcfg-http-ede84d-79fd9b8df5-4r8vk\" (UID: \"c2fabbd7-9e87-4417-9a52-544dc74cba58\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ede84d-79fd9b8df5-4r8vk" Nov 01 23:25:16.501274 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:16.501186 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3123603-33bb-4484-a1b4-dccc87479653-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-ede84d-6c6c4ddd87-722rr\" (UID: \"b3123603-33bb-4484-a1b4-dccc87479653\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-ede84d-6c6c4ddd87-722rr" Nov 01 23:25:16.501274 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:16.501245 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt848\" (UniqueName: \"kubernetes.io/projected/b3123603-33bb-4484-a1b4-dccc87479653-kube-api-access-gt848\") pod \"payload-processing-e2e-authcfg-http-ede84d-6c6c4ddd87-722rr\" (UID: \"b3123603-33bb-4484-a1b4-dccc87479653\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-ede84d-6c6c4ddd87-722rr" Nov 01 23:25:16.502002 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:16.501975 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3123603-33bb-4484-a1b4-dccc87479653-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-ede84d-6c6c4ddd87-722rr\" (UID: \"b3123603-33bb-4484-a1b4-dccc87479653\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-ede84d-6c6c4ddd87-722rr" Nov 01 23:25:16.509494 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:16.509468 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt848\" (UniqueName: \"kubernetes.io/projected/b3123603-33bb-4484-a1b4-dccc87479653-kube-api-access-gt848\") pod \"payload-processing-e2e-authcfg-http-ede84d-6c6c4ddd87-722rr\" (UID: \"b3123603-33bb-4484-a1b4-dccc87479653\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-ede84d-6c6c4ddd87-722rr" Nov 01 23:25:16.519169 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:16.519149 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ede84d-79fd9b8df5-4r8vk" Nov 01 23:25:16.575351 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:16.574109 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ede84d-6c6c4ddd87-722rr" Nov 01 23:25:16.828877 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:16.828854 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-ede84d-79fd9b8df5-4r8vk"] Nov 01 23:25:16.830962 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:25:16.830936 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2fabbd7_9e87_4417_9a52_544dc74cba58.slice/crio-7a125d3b734c2508da48508373479edce9003d8d8ec4be447daba667de9fbdc1 WatchSource:0}: Error finding container 7a125d3b734c2508da48508373479edce9003d8d8ec4be447daba667de9fbdc1: Status 404 returned error can't find the container with id 7a125d3b734c2508da48508373479edce9003d8d8ec4be447daba667de9fbdc1 Nov 01 23:25:16.900524 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:16.900400 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-ede84d-6c6c4ddd87-722rr"] Nov 01 23:25:16.915673 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:25:16.915638 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3123603_33bb_4484_a1b4_dccc87479653.slice/crio-7ab04491a832144569e2b71192d5156c74a7f9332d82ad3c6630948d44cb2e59 WatchSource:0}: Error finding container 7ab04491a832144569e2b71192d5156c74a7f9332d82ad3c6630948d44cb2e59: Status 404 returned error can't find the container with id 7ab04491a832144569e2b71192d5156c74a7f9332d82ad3c6630948d44cb2e59 Nov 01 23:25:16.988394 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:25:16.988348 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:25:17.620582 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:17.620527 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ede84d-6c6c4ddd87-722rr" event={"ID":"b3123603-33bb-4484-a1b4-dccc87479653","Type":"ContainerStarted","Data":"2e7a3bee18642a844192e216bb039971edb894118ab8f718d6c65067a8f6ee6f"} Nov 01 23:25:17.621045 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:17.620605 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ede84d-6c6c4ddd87-722rr" Nov 01 23:25:17.621045 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:17.620630 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ede84d-6c6c4ddd87-722rr" event={"ID":"b3123603-33bb-4484-a1b4-dccc87479653","Type":"ContainerStarted","Data":"7ab04491a832144569e2b71192d5156c74a7f9332d82ad3c6630948d44cb2e59"} Nov 01 23:25:17.622147 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:17.622117 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ede84d-79fd9b8df5-4r8vk" event={"ID":"c2fabbd7-9e87-4417-9a52-544dc74cba58","Type":"ContainerStarted","Data":"3d368da34ca8d855d0981e87822af6ce80af877ef9265c679aca1525c0b5911a"} Nov 01 23:25:17.622299 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:17.622153 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ede84d-79fd9b8df5-4r8vk" event={"ID":"c2fabbd7-9e87-4417-9a52-544dc74cba58","Type":"ContainerStarted","Data":"7a125d3b734c2508da48508373479edce9003d8d8ec4be447daba667de9fbdc1"} Nov 01 23:25:17.622299 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:17.622182 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ede84d-79fd9b8df5-4r8vk" Nov 01 23:25:17.637425 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:17.637380 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ede84d-6c6c4ddd87-722rr" podStartSLOduration=1.637368092 podStartE2EDuration="1.637368092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:25:17.636192327 +0000 UTC m=+3686.040782960" watchObservedRunningTime="2026-11-01 23:25:17.637368092 +0000 UTC m=+3686.041958723" Nov 01 23:25:17.652560 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:17.652518 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ede84d-79fd9b8df5-4r8vk" podStartSLOduration=1.6525053079999998 podStartE2EDuration="1.652505308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:25:17.651969991 +0000 UTC m=+3686.056560624" watchObservedRunningTime="2026-11-01 23:25:17.652505308 +0000 UTC m=+3686.057095940" Nov 01 23:25:18.294124 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:18.294085 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:25:28.632266 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:28.632233 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ede84d-79fd9b8df5-4r8vk" Nov 01 23:25:28.632709 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:28.632303 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ede84d-6c6c4ddd87-722rr" Nov 01 23:25:46.958901 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:25:46.958800 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:25:55.850458 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:55.850425 5011 scope.go:122] "RemoveContainer" containerID="389b6e1027ec1e4c48776c7d5e4bbb9debc43a5fd5f4100f7991e504dace69db" Nov 01 23:25:55.859344 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:25:55.859297 5011 scope.go:122] "RemoveContainer" containerID="63a3b114b7854683f664674f9613585fe5cd22e1a9690f62e2ea59153f075da5" Nov 01 23:26:16.963137 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:26:16.963087 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:26:46.959343 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:26:46.959295 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:27:02.714416 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:02.714322 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-ede84d-6c6c4ddd87-722rr"] Nov 01 23:27:02.727565 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:02.727522 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-ede84d-79fd9b8df5-4r8vk"] Nov 01 23:27:07.715445 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:07.715367 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ede84d-6c6c4ddd87-722rr" podUID="b3123603-33bb-4484-a1b4-dccc87479653" containerName="payload-processing" containerID="cri-o://2e7a3bee18642a844192e216bb039971edb894118ab8f718d6c65067a8f6ee6f" gracePeriod=25 Nov 01 23:27:07.728297 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:07.728228 5011 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ede84d-79fd9b8df5-4r8vk" podUID="c2fabbd7-9e87-4417-9a52-544dc74cba58" containerName="payload-pre-processing" containerID="cri-o://3d368da34ca8d855d0981e87822af6ce80af877ef9265c679aca1525c0b5911a" gracePeriod=25 Nov 01 23:27:07.985939 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:07.985912 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ede84d-6c6c4ddd87-722rr" Nov 01 23:27:08.014354 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:08.014323 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ede84d-79fd9b8df5-4r8vk" Nov 01 23:27:08.124481 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:08.124444 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3123603-33bb-4484-a1b4-dccc87479653-kube-api-access-gt848\" (UniqueName: \"kubernetes.io/projected/b3123603-33bb-4484-a1b4-dccc87479653-kube-api-access-gt848\") pod \"b3123603-33bb-4484-a1b4-dccc87479653\" (UID: \"b3123603-33bb-4484-a1b4-dccc87479653\") " Nov 01 23:27:08.124687 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:08.124523 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3123603-33bb-4484-a1b4-dccc87479653-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3123603-33bb-4484-a1b4-dccc87479653-plugins-config-volume\") pod \"b3123603-33bb-4484-a1b4-dccc87479653\" (UID: \"b3123603-33bb-4484-a1b4-dccc87479653\") " Nov 01 23:27:08.124687 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:08.124637 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2fabbd7-9e87-4417-9a52-544dc74cba58-kube-api-access-dk6bv\" (UniqueName: \"kubernetes.io/projected/c2fabbd7-9e87-4417-9a52-544dc74cba58-kube-api-access-dk6bv\") pod \"c2fabbd7-9e87-4417-9a52-544dc74cba58\" (UID: \"c2fabbd7-9e87-4417-9a52-544dc74cba58\") " Nov 01 23:27:08.124825 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:08.124722 5011 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2fabbd7-9e87-4417-9a52-544dc74cba58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2fabbd7-9e87-4417-9a52-544dc74cba58-plugins-config-volume\") pod \"c2fabbd7-9e87-4417-9a52-544dc74cba58\" (UID: \"c2fabbd7-9e87-4417-9a52-544dc74cba58\") " Nov 01 23:27:08.124916 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:08.124866 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3123603-33bb-4484-a1b4-dccc87479653-plugins-config-volume" pod "b3123603-33bb-4484-a1b4-dccc87479653" (UID: "b3123603-33bb-4484-a1b4-dccc87479653"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:27:08.125120 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:08.125101 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3123603-33bb-4484-a1b4-dccc87479653-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:27:08.125120 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:08.125107 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2fabbd7-9e87-4417-9a52-544dc74cba58-plugins-config-volume" pod "c2fabbd7-9e87-4417-9a52-544dc74cba58" (UID: "c2fabbd7-9e87-4417-9a52-544dc74cba58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:27:08.126865 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:08.126824 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3123603-33bb-4484-a1b4-dccc87479653-kube-api-access-gt848" pod "b3123603-33bb-4484-a1b4-dccc87479653" (UID: "b3123603-33bb-4484-a1b4-dccc87479653"). InnerVolumeSpecName "kube-api-access-gt848". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:27:08.126930 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:08.126866 5011 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2fabbd7-9e87-4417-9a52-544dc74cba58-kube-api-access-dk6bv" pod "c2fabbd7-9e87-4417-9a52-544dc74cba58" (UID: "c2fabbd7-9e87-4417-9a52-544dc74cba58"). InnerVolumeSpecName "kube-api-access-dk6bv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:27:08.190122 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:08.189910 5011 generic.go:356] "Generic (PLEG): container finished" podID="c2fabbd7-9e87-4417-9a52-544dc74cba58" containerID="3d368da34ca8d855d0981e87822af6ce80af877ef9265c679aca1525c0b5911a" exitCode=0 Nov 01 23:27:08.190122 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:08.189958 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ede84d-79fd9b8df5-4r8vk" event={"ID":"c2fabbd7-9e87-4417-9a52-544dc74cba58","Type":"ContainerDied","Data":"3d368da34ca8d855d0981e87822af6ce80af877ef9265c679aca1525c0b5911a"} Nov 01 23:27:08.190122 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:08.190000 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ede84d-79fd9b8df5-4r8vk" Nov 01 23:27:08.190122 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:08.190046 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ede84d-79fd9b8df5-4r8vk" event={"ID":"c2fabbd7-9e87-4417-9a52-544dc74cba58","Type":"ContainerDied","Data":"7a125d3b734c2508da48508373479edce9003d8d8ec4be447daba667de9fbdc1"} Nov 01 23:27:08.190122 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:08.190096 5011 scope.go:122] "RemoveContainer" containerID="3d368da34ca8d855d0981e87822af6ce80af877ef9265c679aca1525c0b5911a" Nov 01 23:27:08.192383 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:08.192257 5011 generic.go:356] "Generic (PLEG): container finished" podID="b3123603-33bb-4484-a1b4-dccc87479653" containerID="2e7a3bee18642a844192e216bb039971edb894118ab8f718d6c65067a8f6ee6f" exitCode=0 Nov 01 23:27:08.192383 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:08.192320 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ede84d-6c6c4ddd87-722rr" event={"ID":"b3123603-33bb-4484-a1b4-dccc87479653","Type":"ContainerDied","Data":"2e7a3bee18642a844192e216bb039971edb894118ab8f718d6c65067a8f6ee6f"} Nov 01 23:27:08.192383 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:08.192356 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ede84d-6c6c4ddd87-722rr" event={"ID":"b3123603-33bb-4484-a1b4-dccc87479653","Type":"ContainerDied","Data":"7ab04491a832144569e2b71192d5156c74a7f9332d82ad3c6630948d44cb2e59"} Nov 01 23:27:08.192636 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:08.192466 5011 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ede84d-6c6c4ddd87-722rr" Nov 01 23:27:08.202710 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:08.202691 5011 scope.go:122] "RemoveContainer" containerID="3d368da34ca8d855d0981e87822af6ce80af877ef9265c679aca1525c0b5911a" Nov 01 23:27:08.203077 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:27:08.203050 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d368da34ca8d855d0981e87822af6ce80af877ef9265c679aca1525c0b5911a\": container with ID starting with 3d368da34ca8d855d0981e87822af6ce80af877ef9265c679aca1525c0b5911a not found: ID does not exist" containerID="3d368da34ca8d855d0981e87822af6ce80af877ef9265c679aca1525c0b5911a" Nov 01 23:27:08.203196 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:08.203091 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d368da34ca8d855d0981e87822af6ce80af877ef9265c679aca1525c0b5911a"} err="failed to get container status \"3d368da34ca8d855d0981e87822af6ce80af877ef9265c679aca1525c0b5911a\": rpc error: code = NotFound desc = could not find container \"3d368da34ca8d855d0981e87822af6ce80af877ef9265c679aca1525c0b5911a\": container with ID starting with 3d368da34ca8d855d0981e87822af6ce80af877ef9265c679aca1525c0b5911a not found: ID does not exist" Nov 01 23:27:08.203196 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:08.203118 5011 scope.go:122] "RemoveContainer" containerID="2e7a3bee18642a844192e216bb039971edb894118ab8f718d6c65067a8f6ee6f" Nov 01 23:27:08.218621 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:08.218561 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-ede84d-79fd9b8df5-4r8vk"] Nov 01 23:27:08.220407 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:08.220357 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-ede84d-79fd9b8df5-4r8vk"] Nov 01 23:27:08.222073 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:08.221638 5011 scope.go:122] "RemoveContainer" containerID="2e7a3bee18642a844192e216bb039971edb894118ab8f718d6c65067a8f6ee6f" Nov 01 23:27:08.222073 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:27:08.222022 5011 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e7a3bee18642a844192e216bb039971edb894118ab8f718d6c65067a8f6ee6f\": container with ID starting with 2e7a3bee18642a844192e216bb039971edb894118ab8f718d6c65067a8f6ee6f not found: ID does not exist" containerID="2e7a3bee18642a844192e216bb039971edb894118ab8f718d6c65067a8f6ee6f" Nov 01 23:27:08.222073 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:08.222059 5011 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e7a3bee18642a844192e216bb039971edb894118ab8f718d6c65067a8f6ee6f"} err="failed to get container status \"2e7a3bee18642a844192e216bb039971edb894118ab8f718d6c65067a8f6ee6f\": rpc error: code = NotFound desc = could not find container \"2e7a3bee18642a844192e216bb039971edb894118ab8f718d6c65067a8f6ee6f\": container with ID starting with 2e7a3bee18642a844192e216bb039971edb894118ab8f718d6c65067a8f6ee6f not found: ID does not exist" Nov 01 23:27:08.226422 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:08.226399 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt848\" (UniqueName: \"kubernetes.io/projected/b3123603-33bb-4484-a1b4-dccc87479653-kube-api-access-gt848\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:27:08.226422 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:08.226423 5011 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk6bv\" (UniqueName: \"kubernetes.io/projected/c2fabbd7-9e87-4417-9a52-544dc74cba58-kube-api-access-dk6bv\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:27:08.226601 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:08.226433 5011 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2fabbd7-9e87-4417-9a52-544dc74cba58-plugins-config-volume\") on node \"ip-10-0-142-238.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:27:08.231891 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:08.231863 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-ede84d-6c6c4ddd87-722rr"] Nov 01 23:27:08.236790 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:08.236769 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-ede84d-6c6c4ddd87-722rr"] Nov 01 23:27:09.746516 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:09.746482 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3123603-33bb-4484-a1b4-dccc87479653" path="/var/lib/kubelet/pods/b3123603-33bb-4484-a1b4-dccc87479653/volumes" Nov 01 23:27:09.746984 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:09.746945 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2fabbd7-9e87-4417-9a52-544dc74cba58" path="/var/lib/kubelet/pods/c2fabbd7-9e87-4417-9a52-544dc74cba58/volumes" Nov 01 23:27:09.786062 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:09.786031 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:27:16.869241 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:16.869209 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:27:16.958799 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:27:16.958763 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:27:21.901671 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:21.901636 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:27:37.178783 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:37.178746 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:27:41.036174 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:41.036134 5011 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f2421d-jnffb"] Nov 01 23:27:41.042705 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:41.042668 5011 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f2421d-jnffb"] Nov 01 23:27:41.770645 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:27:41.770236 5011 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a19decc7-a11d-49de-b391-abd4ce7d6a54" path="/var/lib/kubelet/pods/a19decc7-a11d-49de-b391-abd4ce7d6a54/volumes" Nov 01 23:27:46.958937 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:27:46.958892 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:28:03.981469 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:03.981432 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-54nss"] Nov 01 23:28:04.127500 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:04.127455 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68ccc5d4b4-cs5zt_ac80b35e-3170-4fba-9039-b51ba2f236ec/authorino/0.log" Nov 01 23:28:14.204479 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:14.204446 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-gf7xg_946fc3be-df64-414f-908d-c1d85fceb06f/manager/0.log" Nov 01 23:28:14.813034 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:14.812994 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-66565fcd54-4b495_1c0b5bb7-8733-417a-9981-475611b520df/manager/0.log" Nov 01 23:28:16.258499 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:16.258472 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7587df7975-bq245_cc13d9fa-de9b-4846-b294-7ba3275093ba/maas-api/0.log" Nov 01 23:28:16.959271 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:28:16.959234 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:28:18.804823 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:18.804768 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68ccc5d4b4-cs5zt_ac80b35e-3170-4fba-9039-b51ba2f236ec/authorino/0.log" Nov 01 23:28:19.742932 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:19.742903 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-54nss_5da97fc4-69c1-4cba-b7d6-9a85b49fb23f/limitador/0.log" Nov 01 23:28:20.768781 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:20.768750 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-mvvdl_c511deaf-bfd4-4c7f-a12e-39f8b80a3a61/discovery/0.log" Nov 01 23:28:20.937965 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:20.937936 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9bc8df847-chtf8_199b342c-2f8e-43eb-a1dd-da546aa1059e/kube-auth-proxy/0.log" Nov 01 23:28:21.769249 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:21.769221 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-bxw6l_4688829c-8b38-4d0f-a56c-f20892644b09/istio-proxy/0.log" Nov 01 23:28:21.935386 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:21.935354 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-dxxsr_dd3a9407-a464-4a77-b83a-b0755b5e0409/istio-proxy/0.log" Nov 01 23:28:22.447372 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:22.447344 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-m4nvr_fd9832d7-c187-41b2-bf9d-de25b0ea455a/istio-proxy/0.log" Nov 01 23:28:22.797022 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:22.796945 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-z9x5s_734444e5-d903-47e2-9ed0-d7617fd3b2ed/istio-proxy/0.log" Nov 01 23:28:24.354037 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:24.354005 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-67p7h_cf198325-a260-434e-a504-f73add235e9b/main/0.log" Nov 01 23:28:24.528738 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:24.528701 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-rpcvn_3b7cef02-a6d1-41ba-9bfd-9fe36d40f249/main/0.log" Nov 01 23:28:24.695953 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:24.695919 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-7q99w_643613a9-01d4-4d8e-bfa3-3cc8b7e118b3/main/0.log" Nov 01 23:28:25.022774 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:25.022673 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-474rg_72f1a530-bd82-48d6-a9a7-020eacd1b1f6/main/0.log" Nov 01 23:28:25.179385 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:25.179342 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr_5c42d6ad-2b35-4217-b81e-18e3b79be22e/main/0.log" Nov 01 23:28:26.140332 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:26.140277 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68ccc5d4b4-cs5zt_ac80b35e-3170-4fba-9039-b51ba2f236ec/authorino/0.log" Nov 01 23:28:34.882049 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:34.881943 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-gf7xg_946fc3be-df64-414f-908d-c1d85fceb06f/manager/0.log" Nov 01 23:28:35.441915 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:35.441884 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-66565fcd54-4b495_1c0b5bb7-8733-417a-9981-475611b520df/manager/0.log" Nov 01 23:28:36.658144 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:36.658107 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7587df7975-bq245_cc13d9fa-de9b-4846-b294-7ba3275093ba/maas-api/0.log" Nov 01 23:28:38.879435 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:38.879405 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68ccc5d4b4-cs5zt_ac80b35e-3170-4fba-9039-b51ba2f236ec/authorino/0.log" Nov 01 23:28:39.776250 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:39.776216 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-54nss_5da97fc4-69c1-4cba-b7d6-9a85b49fb23f/limitador/0.log" Nov 01 23:28:40.631728 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:40.631683 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-mvvdl_c511deaf-bfd4-4c7f-a12e-39f8b80a3a61/discovery/0.log" Nov 01 23:28:40.798574 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:40.798541 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9bc8df847-chtf8_199b342c-2f8e-43eb-a1dd-da546aa1059e/kube-auth-proxy/0.log" Nov 01 23:28:41.657377 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:41.657315 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-bxw6l_4688829c-8b38-4d0f-a56c-f20892644b09/istio-proxy/0.log" Nov 01 23:28:41.816492 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:41.816460 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-dxxsr_dd3a9407-a464-4a77-b83a-b0755b5e0409/istio-proxy/0.log" Nov 01 23:28:42.314331 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:42.314304 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-m4nvr_fd9832d7-c187-41b2-bf9d-de25b0ea455a/istio-proxy/0.log" Nov 01 23:28:42.635723 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:42.635626 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-z9x5s_734444e5-d903-47e2-9ed0-d7617fd3b2ed/istio-proxy/0.log" Nov 01 23:28:43.981057 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:43.981029 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-67p7h_cf198325-a260-434e-a504-f73add235e9b/main/0.log" Nov 01 23:28:44.128070 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:44.128036 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-rpcvn_3b7cef02-a6d1-41ba-9bfd-9fe36d40f249/main/0.log" Nov 01 23:28:44.282595 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:44.282468 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-7q99w_643613a9-01d4-4d8e-bfa3-3cc8b7e118b3/main/0.log" Nov 01 23:28:44.639623 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:44.639543 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-474rg_72f1a530-bd82-48d6-a9a7-020eacd1b1f6/main/0.log" Nov 01 23:28:44.804579 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:44.804549 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-d8vcr_5c42d6ad-2b35-4217-b81e-18e3b79be22e/main/0.log" Nov 01 23:28:46.960033 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:28:46.959997 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:28:55.922152 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:55.922122 5011 scope.go:122] "RemoveContainer" containerID="2b79d09ea579505e19456747a8646a647fdfa0bfdb2f49936e403aa3bcdabd43" Nov 01 23:28:55.930620 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:28:55.930599 5011 scope.go:122] "RemoveContainer" containerID="e0c17f57109732eda4dfea0358c0c1ec4d59926f697b7e70bcc44cf00198fb38" Nov 01 23:29:16.960331 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:29:16.960289 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:29:20.363376 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:29:20.363347 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-66565fcd54-4b495_1c0b5bb7-8733-417a-9981-475611b520df/manager/0.log" Nov 01 23:29:20.611960 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:29:20.611930 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-66565fcd54-4b495_1c0b5bb7-8733-417a-9981-475611b520df/manager/0.log" Nov 01 23:29:46.958413 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:29:46.958377 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:30:05.690536 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:05.690457 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-66565fcd54-4b495_1c0b5bb7-8733-417a-9981-475611b520df/manager/0.log" Nov 01 23:30:05.968410 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:05.968328 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-66565fcd54-4b495_1c0b5bb7-8733-417a-9981-475611b520df/manager/0.log" Nov 01 23:30:10.517371 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:10.517338 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ttjsz_fa603801-6c30-4cf1-ace8-30c8723e4a1e/global-pull-secret-syncer/0.log" Nov 01 23:30:10.788574 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:10.788490 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tmhm2_3bed4cac-d251-49be-9eb6-ce5cdb6852de/konnectivity-agent/0.log" Nov 01 23:30:10.907028 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:10.907000 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-238.us-east-2.compute.internal_4f1e77a94f837e0f1b12b6a1e77eab89/haproxy/0.log" Nov 01 23:30:16.960845 ip-10-0-142-238 kubenswrapper[5011]: E1101 23:30:16.960796 5011 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lcqwg" containerName="extractor" Nov 01 23:30:17.972681 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:17.972645 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68ccc5d4b4-cs5zt_ac80b35e-3170-4fba-9039-b51ba2f236ec/authorino/0.log" Nov 01 23:30:18.162872 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:18.162820 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-54nss_5da97fc4-69c1-4cba-b7d6-9a85b49fb23f/limitador/0.log" Nov 01 23:30:19.616784 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:19.616754 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a5e2273-88cc-4c23-a168-c5039e5b7791/alertmanager/0.log" Nov 01 23:30:19.643872 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:19.643822 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a5e2273-88cc-4c23-a168-c5039e5b7791/config-reloader/0.log" Nov 01 23:30:19.673387 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:19.673357 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a5e2273-88cc-4c23-a168-c5039e5b7791/kube-rbac-proxy-web/0.log" Nov 01 23:30:19.702204 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:19.702176 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a5e2273-88cc-4c23-a168-c5039e5b7791/kube-rbac-proxy/0.log" Nov 01 23:30:19.738308 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:19.738283 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a5e2273-88cc-4c23-a168-c5039e5b7791/kube-rbac-proxy-metric/0.log" Nov 01 23:30:19.763911 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:19.763892 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a5e2273-88cc-4c23-a168-c5039e5b7791/prom-label-proxy/0.log" Nov 01 23:30:19.789160 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:19.789136 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a5e2273-88cc-4c23-a168-c5039e5b7791/init-config-reloader/0.log" Nov 01 23:30:19.955456 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:19.955429 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-66f8d6ccdb-f9xnq_ac8d3bc4-33c4-4843-a23b-91c6908a45d1/metrics-server/0.log" Nov 01 23:30:19.983092 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:19.983065 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-h4shp_d9be2a9d-8cd7-4fc4-ab6a-8b3c7793c5bd/monitoring-plugin/0.log" Nov 01 23:30:20.015083 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.015059 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4f58c_ca5a7ca1-bb6d-4e12-90da-e0a1831153b0/node-exporter/0.log" Nov 01 23:30:20.040245 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.040222 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4f58c_ca5a7ca1-bb6d-4e12-90da-e0a1831153b0/kube-rbac-proxy/0.log" Nov 01 23:30:20.063247 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.063226 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4f58c_ca5a7ca1-bb6d-4e12-90da-e0a1831153b0/init-textfile/0.log" Nov 01 23:30:20.553761 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.553726 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-sxspm_e296d10d-5064-4f8f-85f0-e6d1b20f1ab8/prometheus-operator/0.log" Nov 01 23:30:20.571841 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.571814 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-sxspm_e296d10d-5064-4f8f-85f0-e6d1b20f1ab8/kube-rbac-proxy/0.log" Nov 01 23:30:20.676318 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.676282 5011 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7bxbb/perf-node-gather-daemonset-wfjtr"] Nov 01 23:30:20.676891 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.676869 5011 cpu_manager.go:414] "containerMap: removing container" podUID="c2fabbd7-9e87-4417-9a52-544dc74cba58" containerName="payload-pre-processing" Nov 01 23:30:20.677001 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.676892 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2fabbd7-9e87-4417-9a52-544dc74cba58" containerName="payload-pre-processing" Nov 01 23:30:20.677001 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.676908 5011 cpu_manager.go:414] "containerMap: removing container" podUID="b3123603-33bb-4484-a1b4-dccc87479653" containerName="payload-processing" Nov 01 23:30:20.677001 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.676915 5011 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3123603-33bb-4484-a1b4-dccc87479653" containerName="payload-processing" Nov 01 23:30:20.677168 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.677041 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2fabbd7-9e87-4417-9a52-544dc74cba58" containerName="payload-pre-processing" Nov 01 23:30:20.677168 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.677055 5011 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3123603-33bb-4484-a1b4-dccc87479653" containerName="payload-processing" Nov 01 23:30:20.680424 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.680402 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-wfjtr" Nov 01 23:30:20.687799 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.687778 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7bxbb\"/\"openshift-service-ca.crt\"" Nov 01 23:30:20.687991 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.687893 5011 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7bxbb\"/\"kube-root-ca.crt\"" Nov 01 23:30:20.688509 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.688486 5011 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7bxbb\"/\"default-dockercfg-k9lzk\"" Nov 01 23:30:20.690687 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.690662 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7bxbb/perf-node-gather-daemonset-wfjtr"] Nov 01 23:30:20.874457 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.874355 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4a86c7ff-ee29-4c70-ac5e-6de400f54fbc-proc\") pod \"perf-node-gather-daemonset-wfjtr\" (UID: \"4a86c7ff-ee29-4c70-ac5e-6de400f54fbc\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-wfjtr" Nov 01 23:30:20.874457 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.874392 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4a86c7ff-ee29-4c70-ac5e-6de400f54fbc-lib-modules\") pod \"perf-node-gather-daemonset-wfjtr\" (UID: \"4a86c7ff-ee29-4c70-ac5e-6de400f54fbc\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-wfjtr" Nov 01 23:30:20.874742 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.874578 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a86c7ff-ee29-4c70-ac5e-6de400f54fbc-sys\") pod \"perf-node-gather-daemonset-wfjtr\" (UID: \"4a86c7ff-ee29-4c70-ac5e-6de400f54fbc\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-wfjtr" Nov 01 23:30:20.874742 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.874691 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tghc\" (UniqueName: \"kubernetes.io/projected/4a86c7ff-ee29-4c70-ac5e-6de400f54fbc-kube-api-access-9tghc\") pod \"perf-node-gather-daemonset-wfjtr\" (UID: \"4a86c7ff-ee29-4c70-ac5e-6de400f54fbc\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-wfjtr" Nov 01 23:30:20.874742 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.874719 5011 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4a86c7ff-ee29-4c70-ac5e-6de400f54fbc-podres\") pod \"perf-node-gather-daemonset-wfjtr\" (UID: \"4a86c7ff-ee29-4c70-ac5e-6de400f54fbc\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-wfjtr" Nov 01 23:30:20.975423 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.975385 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4a86c7ff-ee29-4c70-ac5e-6de400f54fbc-podres\") pod \"perf-node-gather-daemonset-wfjtr\" (UID: \"4a86c7ff-ee29-4c70-ac5e-6de400f54fbc\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-wfjtr" Nov 01 23:30:20.975624 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.975438 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4a86c7ff-ee29-4c70-ac5e-6de400f54fbc-proc\") pod \"perf-node-gather-daemonset-wfjtr\" (UID: \"4a86c7ff-ee29-4c70-ac5e-6de400f54fbc\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-wfjtr" Nov 01 23:30:20.975624 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.975458 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4a86c7ff-ee29-4c70-ac5e-6de400f54fbc-lib-modules\") pod \"perf-node-gather-daemonset-wfjtr\" (UID: \"4a86c7ff-ee29-4c70-ac5e-6de400f54fbc\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-wfjtr" Nov 01 23:30:20.975624 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.975527 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a86c7ff-ee29-4c70-ac5e-6de400f54fbc-sys\") pod \"perf-node-gather-daemonset-wfjtr\" (UID: \"4a86c7ff-ee29-4c70-ac5e-6de400f54fbc\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-wfjtr" Nov 01 23:30:20.975624 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.975525 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4a86c7ff-ee29-4c70-ac5e-6de400f54fbc-podres\") pod \"perf-node-gather-daemonset-wfjtr\" (UID: \"4a86c7ff-ee29-4c70-ac5e-6de400f54fbc\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-wfjtr" Nov 01 23:30:20.975624 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.975529 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4a86c7ff-ee29-4c70-ac5e-6de400f54fbc-proc\") pod \"perf-node-gather-daemonset-wfjtr\" (UID: \"4a86c7ff-ee29-4c70-ac5e-6de400f54fbc\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-wfjtr" Nov 01 23:30:20.975624 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.975591 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a86c7ff-ee29-4c70-ac5e-6de400f54fbc-sys\") pod \"perf-node-gather-daemonset-wfjtr\" (UID: \"4a86c7ff-ee29-4c70-ac5e-6de400f54fbc\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-wfjtr" Nov 01 23:30:20.975624 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.975590 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4a86c7ff-ee29-4c70-ac5e-6de400f54fbc-lib-modules\") pod \"perf-node-gather-daemonset-wfjtr\" (UID: \"4a86c7ff-ee29-4c70-ac5e-6de400f54fbc\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-wfjtr" Nov 01 23:30:20.975624 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.975616 5011 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tghc\" (UniqueName: \"kubernetes.io/projected/4a86c7ff-ee29-4c70-ac5e-6de400f54fbc-kube-api-access-9tghc\") pod \"perf-node-gather-daemonset-wfjtr\" (UID: \"4a86c7ff-ee29-4c70-ac5e-6de400f54fbc\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-wfjtr" Nov 01 23:30:20.985124 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.985084 5011 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tghc\" (UniqueName: \"kubernetes.io/projected/4a86c7ff-ee29-4c70-ac5e-6de400f54fbc-kube-api-access-9tghc\") pod \"perf-node-gather-daemonset-wfjtr\" (UID: \"4a86c7ff-ee29-4c70-ac5e-6de400f54fbc\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-wfjtr" Nov 01 23:30:20.992822 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:20.992802 5011 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-wfjtr" Nov 01 23:30:21.271329 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:21.271249 5011 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7bxbb/perf-node-gather-daemonset-wfjtr"] Nov 01 23:30:21.274291 ip-10-0-142-238 kubenswrapper[5011]: W1101 23:30:21.274259 5011 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4a86c7ff_ee29_4c70_ac5e_6de400f54fbc.slice/crio-98daccd635b17f83031995f4de21515b1a12368dcc0be6ee8ce4108ae0baa87e WatchSource:0}: Error finding container 98daccd635b17f83031995f4de21515b1a12368dcc0be6ee8ce4108ae0baa87e: Status 404 returned error can't find the container with id 98daccd635b17f83031995f4de21515b1a12368dcc0be6ee8ce4108ae0baa87e Nov 01 23:30:22.232537 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:22.232500 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-wfjtr" event={"ID":"4a86c7ff-ee29-4c70-ac5e-6de400f54fbc","Type":"ContainerStarted","Data":"2e6a3c15e6b247422a604f4b625ca36e5fb6b463aa2d1d7daa64d2774b849082"} Nov 01 23:30:22.232969 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:22.232684 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-wfjtr" Nov 01 23:30:22.232969 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:22.232705 5011 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-wfjtr" event={"ID":"4a86c7ff-ee29-4c70-ac5e-6de400f54fbc","Type":"ContainerStarted","Data":"98daccd635b17f83031995f4de21515b1a12368dcc0be6ee8ce4108ae0baa87e"} Nov 01 23:30:22.252606 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:22.252564 5011 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-wfjtr" podStartSLOduration=2.252551133 podStartE2EDuration="2.252551133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 23:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:30:22.250048248 +0000 UTC m=+3990.654638893" watchObservedRunningTime="2026-11-01 23:30:22.252551133 +0000 UTC m=+3990.657141758" Nov 01 23:30:23.210818 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:23.210786 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-c9c7cf75c-4qs78_bf1d3e40-6dcb-490f-82c4-ac196ff2589d/console/0.log" Nov 01 23:30:25.120864 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:25.120811 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-srdnl_9af8b8a8-47f6-4576-87db-2f0253903e5b/dns/0.log" Nov 01 23:30:25.143389 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:25.143364 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-srdnl_9af8b8a8-47f6-4576-87db-2f0253903e5b/kube-rbac-proxy/0.log" Nov 01 23:30:25.221965 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:25.221935 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x4k8b_f71460db-658f-45ed-980f-da65509e2bc5/dns-node-resolver/0.log" Nov 01 23:30:25.881561 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:25.881535 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rkf2j_682b0c5d-fa6f-4a17-8b72-a1f224cf5fec/node-ca/0.log" Nov 01 23:30:28.250346 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:28.250315 5011 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-wfjtr" Nov 01 23:30:28.739568 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:28.739529 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-mvvdl_c511deaf-bfd4-4c7f-a12e-39f8b80a3a61/discovery/0.log" Nov 01 23:30:29.179742 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:29.179714 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9bc8df847-chtf8_199b342c-2f8e-43eb-a1dd-da546aa1059e/kube-auth-proxy/0.log" Nov 01 23:30:29.347757 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:29.347722 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-bxw6l_4688829c-8b38-4d0f-a56c-f20892644b09/istio-proxy/0.log" Nov 01 23:30:29.378710 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:29.378684 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-dxxsr_dd3a9407-a464-4a77-b83a-b0755b5e0409/istio-proxy/0.log" Nov 01 23:30:29.490569 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:29.490485 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-m4nvr_fd9832d7-c187-41b2-bf9d-de25b0ea455a/istio-proxy/0.log" Nov 01 23:30:29.558678 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:29.558651 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-z9x5s_734444e5-d903-47e2-9ed0-d7617fd3b2ed/istio-proxy/0.log" Nov 01 23:30:32.401155 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:32.401122 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v2l2j_b3a3cc56-6cfb-4092-ba39-5c237ad58c77/serve-healthcheck-canary/0.log" Nov 01 23:30:32.428518 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:32.428490 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-mvvdl_c511deaf-bfd4-4c7f-a12e-39f8b80a3a61/discovery/0.log" Nov 01 23:30:32.438691 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:32.438666 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9bc8df847-chtf8_199b342c-2f8e-43eb-a1dd-da546aa1059e/kube-auth-proxy/0.log" Nov 01 23:30:32.542500 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:32.542455 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-bxw6l_4688829c-8b38-4d0f-a56c-f20892644b09/istio-proxy/0.log" Nov 01 23:30:32.572520 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:32.572487 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-dxxsr_dd3a9407-a464-4a77-b83a-b0755b5e0409/istio-proxy/0.log" Nov 01 23:30:32.642508 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:32.642475 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-m4nvr_fd9832d7-c187-41b2-bf9d-de25b0ea455a/istio-proxy/0.log" Nov 01 23:30:32.693041 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:32.693012 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-z9x5s_734444e5-d903-47e2-9ed0-d7617fd3b2ed/istio-proxy/0.log" Nov 01 23:30:33.007693 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:33.007606 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lcqwg_eb7b37b5-9639-45da-b90e-e08dcfac93f3/kube-rbac-proxy/0.log" Nov 01 23:30:33.030289 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:33.030263 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lcqwg_eb7b37b5-9639-45da-b90e-e08dcfac93f3/exporter/0.log" Nov 01 23:30:33.054013 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:33.053987 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lcqwg_eb7b37b5-9639-45da-b90e-e08dcfac93f3/extractor/0.log" Nov 01 23:30:33.166441 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:33.166406 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-hpj5f-7thv7_ed5e5b02-53ae-4da1-9ed5-43815cb00ce0/insights-gathering/0.log" Nov 01 23:30:35.612181 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:35.612143 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-5b59bbd84f-qqvnv_44fa05ae-24ca-4cc3-954d-47307ffdbfae/manager/0.log" Nov 01 23:30:35.630246 ip-10-0-142-238 kubenswrapper[5011]: I1101 23:30:35.630218 5011 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-5b59bbd84f-qqvnv_44fa05ae-24ca-4cc3-954d-47307ffdbfae/copy-manifests/0.log"